🚀 go-pugleaf

Paul Tukey writes:

> I feel compelled to express a rather different point of view, which some
> may find provocative:

In my view what you say is not inconsistent with what I said, but
complementary, and looking at the matter the other way round.

> I would agree that poor experimental design (in those cases where you
> have the luxury of complete control over the experimental design) can
> produce unnecessary collinearity.
>
> But I would say that in many cases the fact that collinearity is
> troublesome is more than anything a result of how the regression
> results are used.
>
> More particularly, it is the result of excessive emphasis on regression
> coefficients as the output of regression, and especially on the
> tendency to interpret individual regression coefficients (with their
> standard errors) in isolation from the others.
>

By "problem with the design" of course I meant "problem with the structure
of the design matrix".  I find it helpful to use linear algebraic language.

If you do a singular value decomposition of your design matrix X:

	X = U S V'

then U gives a canonical orthonormal basis for the model space, and S and V
give insights on any near collinearities present and their effects.  In
particular V specifies an independent set of linear functions of the
regression coefficient vector, and the corresponding diagonal entry in S^2
gives the relative (Fisher) information in the experiment for each of them.

My remarks were to the effect that if you happen to be interested in linear
functions of V corresponding to the smaller diagonal entries in S^2 you may
have a problem in that your design matrix is poorly placed to estimate them.

I interpret Paul's example as saying that if, fortunately, you happen to be
interested mainly in linear functions of V corresponding to the larger
diagonal entries in S^2, then you may well not have a problem since your
design matrix is well placed to estimate them.

[OK, so I'm a Jeremiah :-)]

> Also, one might consider reviewing the subject of principal components
> regression to clarify further some of these things.

Of course principal component regression is really the same as the singular
value decomposition argument I give above.

> All of my remarks above are in the context of linear regression, but I am
> sure that the basic insights have direct analogies in logistic regression.

Yes.  The only complicating factor in logistic regression is that the
relevant metric in sample space is no longer Euclidean but Riemannian, and
it depends upon the unknown parameters (regression coefficients) rather than
fixed in advance and "flat".  However in practice this often doesn't amount
to much of a difficulty for reasonably large experiments.  If you would have
had a problem with an ordinary regression, then more than likely you will
also have one with logistic regression, and conversely.

[Sorry to go on at length, but this is an important matter to get straight
and to handle correctly.  I will not say any more on this matter, promise!]

--
Bill Venables, Dept. of Statistics, | Email: venables@stats.adelaide.edu.au
Univ. of Adelaide, South Australia. | Tel: +61 8 228 5412  Fax: ...232 5670