🚀 go-pugleaf

juber@uceng.UC.EDU (james uber) writes:


>In article <thompson.714414133@daphne.socsci.umn.edu> thompson@atlas.socsci.umn.edu writes:
>>juber@uceng.UC.EDU (james uber) writes:
>>
>>>I obtain parameter estimates via maximum likelihood where
>>>my model is in the standard reduced form y = f(p), y are the
>>>data and p are the parameters. I assume that the distribution
>>>of the model + measurement errors is normal with zero mean
>>>and known covariance matrix Ve. Thus i am solving the optimization
>>>problem:
>>
>>>	min Tr(y - f(p))Inv(Ve)(y - f(p))
>>>         p
>>
[stuff deleted]
>Thanks for replying to my post. That was my fault for being too 
>hasty. This is what i really meant (now given a second chance).
>The _measurement_ errors are defined as:

>        e1 = y - y*

>where y are the data and y* are the (unknown) true values. Now the
>_model_ errors are defined as:

>        y* = f(p*) + e2

>where p* are the "true" parameter values. That is, even given the
>measurements without error and the true parameters, there still is
>some error due to faults in the model theory, inaccuracy in solution
>of f(p), and the like. Combining these two equations gets me back
>to where i should have started in the beginning:

>        y - e1 = f(p*) + e2
>        y = f(p*) + e1 + e2
>        y = f(p*) + E

>where E = e1 + e2 is the combined model + measurment errors. Thus
>the relevant distribution to use in estimation of p via maximum 
>likelihood is the distribution of E, which i previously assumed
>was a normal distribution with known covariance Ve. My
>understanding is that, while it is often possible to specify the 
>distribution of e1 in a logical way (normal and i.i.d., for example),
>the same can not necessarily be said for e2. Hence the significance
>of knowing that both exist. I realize that the parameters of the
>distribution of E can be estimated, under certain assumptions
>about their form.

>Now, like you said, if f(p) is known (not random), then the
>variability in y comes from the variability in E. This is where my
>brain start to hurt, `cause i think to myself, "hey, wait a minute,
>we're talking about the variability in the _data_, which are the
>_measurement_ errors." Thus i get confused when i look at 
>derivations of the covariance matrix of the parameter estimates
>that say things like, "the maximum likelihood objective function
>depends on the _data_," or "if we vary the _data_ slightly,
>replacing y by y + dy, this would cause the minimum (of the 
>log likelihood function) to shift from p* to p* + dp*." I just
>can't get around thinking of the variability in my data as the
>measurement (e1) errors! So, this leads to my original question
>of, in the derivation of the p covariance estimate, when we talk
>of the variability in p being caused by the variability in y, 
>do we mean variability caused by e1 or by E? I am fairly certain 
>that it must be E, but applied statistics is a tough business to 
>part-timers, and i'm just not sure.

First, a warning.  In your original post, and in the paragraph above,
you use p* as the ML estimate, while in the first part of your
clarification you use p* to represent the true parameter values.  Bad
idea.  I will use p* to represent the "truth" and p^ to represent the
MLE.

In order to discuss "the" variabilility in p^ we must have some notion
of what is the relevant probability space.  That is, we must specify
the distribution of the objects from which p^ is computed.
Specifically, we must commit to whether or not e2 (the "model" error)
is fixed or not.  Equivalently, we must think about the sources of
variability in the data, and decide which ones we are interested in.

Scenario 1.  If we take the view that e1 and e2 are both variable then

         var(y) = var( f(p*) + e1 + e2 ) = var( e1 + e2 ) = var(E)

(assuming here and throughout that f(p*) is fixed).  Clearly the
distinction between e1 and e2 is not relevant for determining the
variablity in p^ in this scenario since they always enter as a sum.
All that matters is Var(E).

I think your confusion arises because you (quite reasonably) see a
fundamental distinction between e1 and e2 that makes Scenario 1
unpalatable.  I suspect that you are thinking about e1 (measurement
error) that is variable across repetitions of the experiment, while e2
(model error) remains fixed.  This is reasonable if e2 in fact arises
because of imperfections in our ability to write down the correct form
for f(p), rather than because of some kind of noise in the
instruments, for example.

Scenario 2.  Assume e2 is fixed (but unknown) across repeated
experiments.  In this case you can calculate the variability in the
data as

        var(y) = var( f(p*) + e1 + e2 ) = var(e1)

because now (in repeated experiments using the same imperfect model)
the model errors, like f(p*), are also fixed.  (This is essentially
the same as we would get by working with conditional (on e2)
distributions in Scenario 1.)

Unfortunately, if e2 is fixed and unknown, the model is not
identified, and you can't estimate p*.  In particular, the MLE
estimator p^ that solves the original minimization problem generally
is not a consistent estimator of p*.  In this case, all of the usual
formulas for discussing the variablity of p^ either (a) break down
and/or (b) are no longer of much relevance.

In other words, you can't learn much about p* unless you know quite a
bit about what you don't know about the model.

Sounds like a Catch-22?  You bet.  The bottom line of most statistical
inference problems is that you have to know something already in order
to learn something else.  If you allow for fixed model errors such as
e2, but are unwilling to say anything about their properties then you
are really saying that you don't know anything at all about f(p*).

Hope this helps.
-- 
T. Scott Thompson              email:  thompson@atlas.socsci.umn.edu
Department of Economics        phone:  (612) 625-0119
University of Minnesota        fax:    (612) 624-0209
--
T. Scott Thompson              email:  thompson@atlas.socsci.umn.edu
Department of Economics        phone:  (612) 625-0119
University of Minnesota        fax:    (612) 624-0209