🚀 go-pugleaf

Last year, the discussion of spreadsheets for statistical computing evolved
into a discussion of collinearity in regression, and now the subject
appears again.  Clearly, this is one of the most troublesome aspects
of regression in practice!

I feel compelled to express a rather different point of view, which some
may find provocative:

Lary Jones writes:

> Bill Venables said:
> > Further to what Dick Darlington has said, with which I concur, I would
> > add the following.
> >
> > Multicollinearity is more than anything a weakness in the *design* of a
> > regression experiment rather than any flaw in the model.
> >
>
> I concur, and IMHO wish to emphasize a slightly different
> point.  It seems that often it is not the "... *design* of a
> regression experiment...", per se, that is at fault.  It is the
> inadequacy of one's measurement tools, and/or the lack of
> independence in the process of measuring the phenomena under
> study.

I would agree that poor experimental design (in those cases where you
have the luxury of complete control over the experimental design) can
produces unnecessary collinearity.

But I would say that in many cases the fact that collinearity is
troublesome is more than anything a result of how the regression
results are used.

More particularly, it is the result of excessive emphasis on regression
coefficients as the output of regression, and especially on the
tendency to interpret individual regression coefficients (with their
standard errors) in isolation from the others.

Let me illustrate this with an example.  In pharamacology, the proper
dosage for certain drugs is proportional to the surface area of the body.
Studies have shown that log surface area can be very well approximated
as a linear function of log weight and log height.  Good values for the
regression coefficients have been published.

Let us suppose that we now have a situation in which we can get direct
measurements of people's weights, but must get their heights from
photographs, and we discover that some people wear hats and others have
elaborate coiffures, making it difficult to find the tops of their
heads in the photographs.  So we cleverly decide to measure the heights
of their eyes from the floor -- surely height of eyes is a good
surrogate for person's height!  Unable to decide between the right
and left eyes, we lazily decide to measure the heights of both.

To calibrate the new procedure, we now collect some data on a
"training sample", and naively regress  Y=(log surface area) on
X1=(log height), X2=(log height of right eye) and X3=(log height of
left eye).  We are enlightened enough (by last year's discussion on
STAT-L) about the dangers of spreadsheets that we don't just invert
the X'X matrix in Lotus 1-2-3, but rather we invoke a reasonably good
statistical package that gives us stable results and doesn't blow
itself out of the water.  Nevertheless, it tells us in no uncertain
terms that the standard errors for the coefficients for X2 and X3 are
very large.

Do we now give up in despair???  No!  What's happening very dramatically
here (and happens in many other situations, but in disguised form) is that
two of our predictor variables (the X's) are measuring THE SAME THING.
It's not that our regression is bad;  indeed, our multiple R-square, etc
is very very good.  It is an excellent regression, and gives very good
prediction.  But because X2 and X3 are largely measuring the same thing,
there is a trade-off in their two fitted coefficients:  I could increase
one of them by .34 and decrease the other by the same amount (probably)
and get an equally good regression fit!  Or I could double one of them and
force the other to zero (roughly equivalent to dropping one of them
from the fit).  Or I could force the two of them to the same value
(the average of the two fitted values), corresponding to starting off
with the *average* eye height in the first place.

Although the individual coefficients for X2 and X3 are poorly determined
(in terms of SE) when taken in isolation, jointly the two are very well
determined, modulo the trade-off between them!  Although the individual
confidence intervals are large, THE JOINT CONFIDENCE ELLIPSE FOR THE TWO
OF THEM IS VERY SMALL (i.e. has very small area, even though it is
stretched in an oblique direction).

The *true* confidence-region output from a regression is not a series
of independent confidence intervals on the individual coefficents,
but a joint conficence ellipsoid (or some other shape, depending
on what kind of fitting algorithm is involved).  Only in the case
where the X'X matrix is diagonal (i.e. the X's are *orthogonal* --
note I say "orthogonal" and not "independent", since there is no
need for them to be regarded as random quantities) are the axes of
the joint confidence ellipsoid lined up with the X axes, allowing
one to safely interpret the CIs for the individual X's in isolation!

Another way of saying this (geometrically) is that the regression
of Y on the subspace spanned by X2 and X3 (in the presence of X1, of course)
is VERY WELL DETERMINED, even though the basis for representing that
subspace is rather arbitrary.

There is a direct connection here with the part of "General Linear
Model Theory" that talks about "estimable functions of the
coefficients".  In our case of eye heights, only one linear function of
the 2 log-eye-heights is "estimable" -- it doesn't much matter here
which one you take -- and the orthogonal linear function of them is
essentially "noise" in this problem.

So, I would say that very often in retrospective or observational
studies (as opposed to designed experiments), collinearity does't
necessarily invalidate the results, but is a phenomenon that raisses
a red flag and tries to tell you to exercise special care in
interpreting one part of the regression output -- the fitted coefficients.
It may, in fact, be urging you to use fewer X's in your fitted model,
or more generally to take "this particular set of X's" less seriously
as "the God-given ones", and to consider combining, reexpressing, and
possibly weeding out some of them.

This point of view is entirely consistent with Dick Darlington's
contribution (since I started writing this) regarding ill-conditioning,
tolerances, and condition numbers.

Also, one might consider reviewing the subject of principal components
regression to further clarify some of these things.

All of my remarks above are in the context of linear regression, but I
am sure that the basic insights have direct analogies in logistic
regression.

Cheers!