Joining the Fun; A Ramble on Parameters

There a very interesting pair of posts and a long thread of insightful comments relating to parameters, both the empirical support for such as well as their suitability given current theoretical commitments.  Cederic, commenting on Neil’s initial post and then adding a longer elaboration, makes the point that nobody seems committed to parameters in the classical sense anymore. Avery and Alex C comment that whatever the empirical shortcomings of parameteric accounts, something is always better than nothing so they reasonably ask what do we replace it with. Alex D rightly points out that the success of parametric accounts is logically independent of the POS and claims about linguistic nativism. In this post, I want to reconstruct the history of how parameter theory arose so as to consider where we ought to go from here. The thoughts ramble on a bit, because I have been trying to figure this out for myself.  Apologies ahead of time.

In the beginning there was the evaluation metric (EM), and Chomsky looked on his work and saw that it was deficient.  The idea in Aspects was that there is a measure of grammatical complexity built into FL and that children in acquiring their I-language (an anachronism here) chose the simplest one compatible with the PLD (viz. the linguistic data available to and used by the child). EM effectively ordered grammars according to their complexity. The idea riffed on ideas of minimal description length around at the time but with the important addition that the aim of a grammatical theory with aspirations of explanatory adequacy was to find the correct UG for specifying the meta-language relevant to determining the correct notion of “description” and “length” in minimal description length. The problem was finding the right things to count when evaluating grammars. At any rate, on this conception, the theory of acquisition involved finding the best overall G compatible with PLD as specified by EM.  Chomsky concluded that this conception, though logically coherent, was not feasible as a learning theory, largely because it looked to be computationally intractable.  Nobody had (nor I believe, has) a good tractable idea of how to compare grammars overall so as to have a complete ordering. Chomsky in LSLT developed some pair-wise metrics for the local comparison of alternative rules, but this is a long way from having a total ordering of the alternative Gs that is required to make EM accounts feasible.  Chomsky’s remedy for this problem: divorce language acquisition from the evaluation of overall grammar formats.

The developments of the Extended Standard Theory, which culminated in GB theories, allows for an alternative conception of acquisition, one that divorces it from measuring overall grammar complexity.  How so? Well, first we eliminated the idea that Gs were compendia of constructions specific rules. And second, we proposed that UG consists of biologically provided schemata (hence part of UG and hence not in need of acquisition) that specify the overall shape of a particular G. On this view, acquisition consists in filling in values for the schematic variables.  Filling in values of UG specified variables is a different task from figuring out the overall shape of the grammar and, on the surface at least, a far more tractable task. The number of parameters being finite already distinguished this from earlier conceptions. In the earlier EM view of things there was no reason to think that the space of grammatical possibilities was finite. Now, as Chomsky emphasized, within a parameter setting model, the space of alternatives, though perhaps very large, was still finite and hence the computational problem was different in kind from the one lightly limned in Aspects. So, divorcing the question of grammatical formats (via the elimination of rules or their reduction to a bare minimum form like ‘move alpha’) from the question of acquisition allowed for what looked like a feasible solution to Plato’s Problem. In place of Gs being sets of constructions specific rules with EMs measuring their overall collective fitness, we had the idea that Gs were vectors of UG specified variables with two possible values (and hence “at most” 2ⁿ possible grammars, a finite number of options). Finding the values was divorced from evaluating sets of rules and this looked feasible.

Note that this is largely a conceptual argument. There is a reasonable hunch but no “proof.” I mention this because other conceptual considerations (we will get to them) can serve to challenge the conclusion and make parameter theories less appealing.

In addition to these conceptual considerations, the comparative grammar research in the 70s, 80s, and 90s provided wow-inducing empirical confirmation of parameter based conceptions. It is hard for current (youngish) practitioners of the grammatical dark arts to appreciate how exciting early work on parameter setting models was. There were effectively three lines of empirical support.

1.     The comparative synchronic grammar research. For example:

a.     The S versus S’ parameter distinguishing Italian from English islands (Rizzi, Sportiche, Torrego)).

b.     The pro drop parameter (correlating, null subjects, inversion and long movement apparently violating the fixed subject/that-t condition (Rizzi, Brandi and Cordin)).

c.     The parametric discussions of anaphoric classes (local and long distance anaphors (Wexler, Borer), to name just three, all uncovered a huge amount of new linguistic data and argued for the fecundity of parametric thinking.

2.     Crain’s “continuity thesis,” which provided evidence that kids “mistakes” in acquiring their particular Gs all actually conform to actual adult Gs. This provided evidence that the space of G options was pretty circumscribed, as a parameter theory implies it is.

3.     The work on diachronic change by Kroch, Lightfoot, Roberts (and more formal work by Berwick and Niyogi) a,o., which indicated that large shifts in grammatical structure over time (e.g. SOV to SVO) could be analyzed as changes in a small number of simple parameter changes.

So, there was a good conceptual reason for moving to parameter models of UG and the move proved to be empirically very fecund. Why the current skepticism?  What’s changed?

To my mind, three changes occurred. As usual, I will start with the conceptual challenges and then proceed to the empirical ones.

The first one can be traced to work first by Dresher and Kaye, and then taken up and further developed with great gusto by Fodor (viz. Janet) and Sakas. This work shows that finite parameter setting can present tractability problems almost as difficult as the ones that Chomsky identified in his rejection of EM models.  What this work demonstrates is that given current envisioned parameters, parameter setting cannot be incremental. Why not? Because parameter values are not independent.  In other words, the value of one parameter in a particular G may depend crucially on that of another. Indeed, the value of any may depend on the value of each and this makes for an explosive combinatory problem. It also makes incremental acquisition mysterious; how do the parameter values get set if any bit of later PLD can completely overturn values previously set?

There have been ingenious solutions to this problem, my favorite being cue-based conceptions (developed by Dresher, Fodor, Lightfoot a.o.). These rely on the notion that there is some data in the PLD that unambiguously determines the value of a parameter. Once set on the basis of this data, the value need never change.  Triggers effectively impose independence on the parameter space. If this is correct, then it renders UG yet more linguistically specific; not only are the parameters very linguistically specific, but the apparatus required to fix these is very linguistically specific as well. Those that don’t like linguistically parochial UGs should really hate both parameter theories and this fix to them. Which brings us to the second conceptual shift: Minimalism.

The minimalist conceit is to eliminate the parochialism of FL and show that the linguistically specific structure of UG can be accounted for in more general cognitive/computational terms. This is motivated both on general methodological grounds (factoring out what is cognitively general from what is linguistically specific is good science) and as a first step to answering Darwin’s Problem, as we’ve discussed at length in other posts. FL internal parameters are a very big challenge to this project. Why? Because UG specified parameters encumber FL with very linguistically specific information (e.g. it’s hard to see how the pro drop parameter (if correct) could possibly be stated in non linguistically specific terms!).

This is what I meant earlier when I noted that conceptual reasons could challenge Chomsky’s earlier conceptual arguments.  Even if parameters made addressing Plato’s Problem more tractable, they may not be a very good solution to the feasibility problem if they severely compromise any approach to Darwin’s. This is what motivates Cederic’s concerns (and others, e.g. Terje Lohndal) I believe, and rightly so.  So, the conceptual landscape has changed and it is not surprising that parameter theories have become less appealing and so open to challenge.

Moreover, as Cederic also stresses, the theoretical landscape has changed as well. A legacy of the GB era that has survived into Minimalism is the agreement that Gs do not consist of construction based rules. Rather, there are very general operations (Merge) with very general constraints (e.g. Extension, Minimality) that allow for a small set of dependencies universally.  Much of this (not all, but much) can be reanalyzed in non linguistically specific terms (or so I believe). With this factored out, there are featural idiosyncracies located in demands made by specific lexical items, but this kind of idiosyncracy may be tolerable as it is segregated to the lexicon, a well known repository of eccentrics.[1]  At any rate, it is easy to see what would motivate a reconsideration of UG internal parameters.

The tractability problems related to parameter setting noted by Dresher-Fodor and company simply add to these motivations. 

That leaves us with the empirical arguments. These alone are what make parameter accounts worth endorsing, if they are well founded, and this is what is currently up for grabs and way beyond my pay grade. Cederic and Fritz Newmeyer (among others) have challenged the empirical validity of the key results. The most important discoveries amounted to the clumping of surface effects with the settings of single values, e.g. pro drop+subject inversion+no that-t effects together as a unit. Find one, you find them all.  However, this is what has been challenged. Is it really true that the groupings of phenomena under single parameter settings is correct?  Do these patterns coagulate as proposed? If not, and this I believe is Newmeyer’s point and strongly empahasized by Cedric, then it is not clear what parameters buy us.  Yes I-languages are different. So? Why think that this difference is due to different parameter settings? So, there is an empirical argument: are there data groupings of the kind earlier proposals advocated? Is the continuity thesis accurate and if so how does one explain this without parameters? These are the two big empirical questions and it is likely to be where the battle over parameters has been joined and, one hopes, will ultimately get resolved.

I’d like to epmpahsize that this is an empirical question.  If the data falls on the classical side then this is a problem for minimalists and exacerbates our task of addressing Darwin’s problem. So be it. Minimalism as I understand it has an empirical core and if it turns out that there is richer structure to UG than I would like, well tough cookies on me (and you if your sympathies tend in the same direction)!

Last point and I will end the rambling here. One nice feature of parameter models is the pretty metaphor it afforded for language acquisition as parameter setting. The switch box model is intuitive and easy to grasp. There is no equivalent for EM models and this is partly why nobody knew what to do with the damn thing.  EM never really got used to generate actual empirical research the way parameter setting models did, at least not in syntax. So can we envision a metaphor for non parameter setting models. I think we can. I offered one in A theory of syntax that I’d like to try and push it again here (I know that this is self aggrandizing, but tooting one’s own horn can be so much fun).  Here’s what I said there (chapter 7):

Assume for a moment that the idea of specified parameters is abandoned. What then?  One attractive property of the GB story was the picture that it came with.  The LAD was analogized to a machine with open switches.  Learning amounts to flipping the switches ‘on’ or ‘off’.  A specific grammar is then just a vector of these switches in one of the two positions.  Given this view there are at most 2^P grammars (P=number of parameters).  There is, in short, a finite amount of possible variation among grammars.

            We can replace this picture of acquisition with another one.  Say that FL provides the basic operations and conditions on their application (e.g. like minimality).  The acquisition process can now be seen as a curve fitting exercise using these given operations.  There is no upper bound on the ways that languages might differ though there are still some things that grammars cannot do.  A possible analogy for this conception of grammar is the variety of geometrical figures that can be drawn using a straight edge and compass.  There is no upper bound on the number of possible different figures.  However, there are many figures that cannot be drawn (e.g. there will be no triangles with 20 degree angles).  Similarly, languages may contain arbitrarily many different kinds of rules depending on the PLD they are trying to fit.

So think of the basic operations and conditions as the analogues of the straight edge and compass and think of language acquisition as fitting the data using these tools. Add to this a few general rules for figure fitting: add a functional category if required, pronounce a bottom copy of a chain rather than a top copy, add an escape hatch to a phase head. These are general procedures that can allow the LAD to escape the strictures of the limited operations a minimalistically stripped down FL makes available.  The analogy is not perfect. But the picture might be helpful in challenging the intuitive availability of the switch box metaphor.

That’s it. This post has also been way too long. Kudos to Neil and Cedric and the various very articulate commenters for making this such a fruitful topic for thought, at least for me. 

---

[1] Though I won’t discuss this now, it seems to me that the Cartographic Project and its discovery of what amounts to a universal base for all Gs is not so easily dismissed. The best hope is to see these substantive universals explicated in semantic terms, not something I am currently optimistic will soon appear.

Grammatical Zombies

The talk on the street is that linguists should stuff their portfolios with probabilistic grammars as they have intrinsic advantages over the old, stogy under-performing non-probabilsitic grammars of yore.  Partisans assure us that going probabilistic makes the learning problem easier, reduces the need for rich innate structures, and even lowers cholestoral levels. Though I am genetically inclined to skepticism (especially about things that sound too good to be true), debunking these claims has always required talents above my pay grade.  Lucky for me I know people who understand these issues deeply and are able to translate their significance for the unwashed, i.e. me. This is all preamble for the following post by Bob Berwick, someone that I have been able to persuade to post on an occasional basis, (hopefully not too occasionally) especially on technical topics from computational linguistics. The bottom line is that ideas that are too good to be true aren't, though their intellectual half lives can be very very long, as Bob explains below. Btw, I stole the title for this post from Bob's apposite reference to John Quiggin, an Australian economist who has done yeoman's work debunking hard to kill bad ideas in macroeconomics.

                                                             Going off the Gold Standard

                                                                     Robert C. Berwick

You don't have to wander very far along the computational linguistics byways these days to stumble across what economist John Quiggin dubs “zombie ideas” - bad memes like “supply-side economics after the Clinton boom and the Bush bust – repeatedly refuted with evidence and analysis" yet somehow always rising from the dead.  Among these, two stand out, virtually articles of probabilistic faith: One, that Gold’s (1967) celebrated results about the non-learnability of most language families given positive-only evidence become moot, once one “goes probabilistic.”  And two, that “going probabilistic” can be cashed out simply by adding probabilities to context-free rules, rendering probabilistic context-free grammars easier to learn than their context-free counterparts.  Easy enough, in fact, to drive a stake through the heart of Demon Rationalism (aka, prior constraints on grammars).  But is any of this true?  Or is it instead, to quote the now-immortal words of Meygn Kelly on election night, just “math you do as a Republican to make yourself feel better”?

Well, Yes and No.  Yes, in that nearly every thoughtful scholar of language learnability, starting just a few years post Gold-rush, bound up in the foundational work of Jay Horning, Jerry Feldman, and Ken Wexler, in the late 1960s, and up to the present day, has stressed that the tough Gold standards for learning exact identification of languages on all possible positive texts – are cognitively implausible and overly restrictive, so should be relaxed in favor of what's called “probabilistic example presentation” and “probabilistic convergence.”  That was, and still is, the consensus view.  In other words, from the very beginning of serious research into language learnability for natural languages, the probabilistic view has stood front and center.  All agree we’re after a learning standard that ensures much more: feasibility. (Chomsky's words, 1965, pp. 53-54) or what Ken Wexler (1980) termed “easy learnability”, that is, “learnability from fairly restricted primary data, in a sufficiently quick time, with limited use of memory” (p. 18).[1] [2]

In probabilistic presentation, a learner gets sentences drawn from some underlying distribution, and only has to succeed on positive example sequence that it gets with sufficient probability, rather than on all positive example sequences, as with Gold.  And in fact, there’s a technical sense in which this move does the trick.  When learnability on all texts – the Gold standard – is relaxed in favor of what Ken Wexler was first to call “measure-1 learnability” at a Stanford workshop in September, 1970, then, unsurprisingly, moving the goalposts changes the game.  What perhaps is surprising is this change seemingly moves the goalposts almost to the line of scrimmage: now, instead of no interesting language classes being learnable, all interesting language classes are learnable (officially, all recursively enumerable language families).

Why?  Picture sentences being spit out one at a time by some underlying stochastic source (aka, the “I blame the parents” model), so every sentence occurs with some probability. (Assuming independent draws – what’s technically called “i.i.d” – draws from an independent and identically distributed stochastic source).  In that case, a learner cannot count on using some vanishingly rare example sentence as a red flag for their correct target language. So, we don’t want to insist, as Gold did, that the learner must succeed on all texts.  If figuring out that your language is English means having to wait for the one David Foster Wallace sentence that begins “Uninitiated adults who might...” and trundles to a close 90 words later with, “subdued, almost narcotized-looking,” then you know you’re goose is cooked.  However, if a learner knows that all longer and longer sentences always become less and less likely, then it can know with high confidence that e.g., the first 100 sentences are more than enough to sort out what particular target language your source has been pitching.

And this is exactly what Horning (1969) did.  He attached probabilities to context-free grammar rules, to get probabilistic context-free grammars (PCFGs).  So for instance, if there are 2 rules in a grammar expanding NPs, NP → Det N and NP → Pronoun, then we have a probability of 0.5 for each.  Since every rule in a PCFG is assumed to be statistically independent, the probability of any particular sentence is simply the product of the probabilities of the rules deriving it.  As sentences get longer and longer, their probabilities must shrink – they become exponentially less likely. If the learner knows this in advance, then it knows that very long sentences are so unlikely that they can be effectively eliminated, with the problem reduced to essentially that of learning to identify a family of finite languages. But we already knew that finite language families are learnable from positive-only examples.[3]

In fact, as Horning (1969, 80-81) suggests, one need not rely on any conventional probability distribution over context-free grammars and languages; one could substitute, for example, a measure that ascribes exponentially lower probabilities to larger grammars, and search for target grammars in order of increasing length, or even a probability measure that just assigns the value 2^–i to the grammar enumerated at step i, G_i (as discussed by Ken Wexler, 1980, fn. 22, 529-532; this is one way to develop the notion of an evaluation metric, something we’ll consider in later posts).  More generally, any so-called uniformly computable probability measure works (as proved in Dan Osherson, Michael Stob, and Scott Weinstein’s book, 1986, chapter 10 pp. 187-188, and by Angluin 1988).

So, are we done? Does adding probabilities to context-free rules really make them easier to learn?  No, there's a catch. TANSTAAFL (viz. There ain’t no such thing as a free lunch!). The measure-1 result holds scant cognitive interest (though it can be said to point the way to what constraints we really do need for human language learning models).  Why? The positive results for measure-1 learnability only work if the learner has extremely strong prior knowledge about the probability distribution that's feeding it sentences.  If you really want the learner’s tabula to be a little more rasa, then as the late Partha Niyogi put the matter, “when one considers statistical learning...all statistical estimation algorithms worth their salt are required to converge in a distribution-free sense” (Niyogi, 2005, p. 67).  So that’s to say, suppose the learner doesn’t have any pre-conceived (and accurate) notion of how the example sentences will be distributed. Problem is, Niyogi then shows, as per Angluin (1988), that distribution-free statistical learning collapses back to Gold learning: if a family of languages is measure-1 learnable in a distribution-free sense, then it is Gold learnable.  Measure-1 learnability doesn't enlarge the class of learnable languages one jot. Oops.

As if that weren’t enough, the assumptions blow away any remaining cognitive fidelity we thought we had won.  For one thing, the statistical independence of PCFG rules is precisely what has been explicitly and strongly rejected by the same folks who embrace Horning, in the recent flood of parsers statistically trained in a supervised way from pre-parsed corpus data.  Rather, such rules are plainly dependent: once you decide a sentence is NP VP, then it’s far more likely that the NP, being a subject, will be expanded as a pronoun, with the reverse true if the NP’s an object – which is just to say that PCFG rules aren’t really statistically independent.  The assumption that sentences are pitched to the learner in i.i.d fashion is plainly wrong.  Nobody believes parents speak this way.

Then too, Horning’s method for selecting grammars – explicit enumeration along with search using Bayes’ rule to hunt for the highest posterior probability grammar given the examples, sometimes collapsing phrase names together and sometimes splitting them apart – proved to be computationally intractable.  (It’s also not a wit different conceptually from what’s been advanced by contemporary Bayesian enthusiasts as far as anyone can tell.) Horning himself remarks on how slow his “550 line Lisp program,”  “EVALUATER”, ran, in part because “the enumerative problem is immense” (pp. 199-120). It could only (partly) find grammars with 2 or 3 nonterminals. To be sure, computation speed has increased by leaps and bounds over the past 35 years from Lisp/36 running on an IBM 360/67. And computer scientists have come up with faster methods to solve Bayesian problems, often approximate – ‘variational Bayes’ and the like.  Nevertheless, the immense space of possible PCFGs – larger than the space of CFGs – confounds search methods to this day.

So in truth, Horning’s enumerative method doesn’t nearly go far enough. PCFGs don’t turn out to be any “easier to learn” than their non-probabilistic counterparts –  perhaps what’s meant here is that it’s  more straightforward to use known techniques to estimate PCFG probabilities.  However, that’s not the same thing as learning the underlying rules themselves.  In fact, estimating language distributions is harder in principle that estimating languages simpliciter, as Niyogi (2005, p.68) observes.  What’s needed, rather, is far, far, stronger medicine, something along the lines of Ken Wexler's “bounded degree of error” approach, in which the possible mistakes a learner can detect with respect to sentences of at most 2 embeddings spans the entire space of all possible errors across a complete family of transformational grammars.  Crucially, Ken’s approach is not enumerative.  Instead, it carries out a search in a tightly-bounded, finite space, with each learning step incrementally improving a single transformational component, rather than selecting and throwing away wholesale entire grammars. But to get this to work, Wexler found out that one must impose far stronger a priori constraints on the family of grammars or languages. 

In our internet era where it sometimes feels as though a veil of ignorance has been drawn across any work that was accomplished before 1990, it is well worth remembering that by that September 1970 workshop held at Stanford, Wexler and colleagues had already set up this model for learning transformational generative grammar where both presentation and convergence were probabilistic, in just the sense described by modern day probability enthusiasts – and found this was not nearly enough.  One had to include constraints on grammars, constraints that wound up to be empirically attested, like locality constraints on how far one could displace phrases.  That work, establishing a positive learnability result, still stands – though rarely cited by statistical enthusiasts despite its probabilistic underpinnings.  In fact, it’s apparently even been recently revived just this past year, in work by Mark Steedman and colleagues (2012), though without any apparent tip of the hat in Ken’s direction.  Mark uses combinatory categorial grammar instead of transformational grammar, but in many other ways follows Ken's model, with the learner using (meaning, surface string) pairs.  Plus ça change.

Next: Why Revolutionary new ideas do appear infrequently.

---

[1]A partial list of people embracing this view would include Jay Horning (1969); Henry Hamburger and Ken Wexler (1970, 1973, 1975); Bob Matthews (1979); Peter Culicover and Ken (1980); yours truly (1982/1985); Dana Angluin (1988); and Partha Niyogi (2005), among others. Here's Horning, 1969, 19, fn. 1, “In the sequel, when we prove identifiability in the limit with text presentation, it is with a different performance requirement, and a different condition on the presentation.”

[2]Here is the quote from Wexler's presentation at a September, 1970 workshop held at Stanford p.159 “A language class is identifiable in the limit with probability 1 with respect to a probabilistic presentation scheme if there exists a learning procedure such that for any member of the class there is a subset of measure 1, of the set of presentation sequences for which the procedure identifies in the limit the language."

[3]In fact, Horning, in what seems to have gone largely unnoticed by statistical enthusiasts stresses several times that his results hold only for the case of unambiguous context-free grammars: 1969: 32-33: “In the sequel we assume...that unambiguous grammars are desired, and will reject grammars which make any sample string ambiguous.”  Of course, on the assumption that natural languages are ambiguous, this immediately makes his results unacceptable from the standpoint of cognitive fidelity.  It’s hard to know just why this has been overlooked for 45 years.  Putting aside the possibility that the people citing Horning in fact haven't really read his thesis, luckily, this omission is of little importance, because Horning simply didn’t know how to place a properly-defined probability measure on the languages generated by ambiguous context-free grammars, a matter that has since been remedied by Osherson, Stob, and Weinstein (1986), who established a far more general result encompassing Horning’s, as noted below.