🚀 go-pugleaf

feh@duke.edu (Frank Harrell) writes:

>Backward elimination does not at all dominate full model fitting.
>See:

[references deleted.]

I am not quite sure what you mean by "full model fitting".  (I can't
check your references right away to find out, since here at Minnesota
the various statistical journals are scattered across two cities, and
I would have to visit possibly three libraries to find them all!)

I will take it to mean "using everything", i.e. running OLS on the set
of all possible regressors, dealing with the colinearities that arise
in some appropriate fashion.  I hope this isn't far off.

I looked again at "Introduction to the Theory and Practice of
Econometrics" (Judge et. al.) that I mentioned in my earlier post.
You are correct that backwards elimination (a form of pretest
estimator) does not dominate the "use everything" strategy.  However,
full model fitting does not dominate backwards elimination either,
provided the "elimination criteria" in the latter is chosen
appropriately.  Neither dominates the other.

I wouldn't want to push this too hard, since the default elimination
criteria implemented in most software is unlikely to be appropriate
except by coincidence.  As I said originally, I mostly sympathize with
the opinion that stepwise regression is "the work of the Devil."

The same chapter (20) also compares shrinkage estimators to the "full
model" approach, and cites evidence that a "Stein-like" shrinkage
estimator does dominate "full model fitting," rendering the latter
inadmissible. It is not clear from this book whether or not backwards
elimination is also dominated by the shrinkage estimator.

(I have always thought of pretest estimators as a randomized form of
shrinkage estimation, hence my confusion here.)

The textbook cites as a source for proofs of some of these assertions:

Judge, G.G., and M.E. Bock (1978) *The Statistical Implications of
Pre-Test and Stein-Rule Estimators in Econometrics*.  Amsterdam:
North-Holland.

I don't think that we can resolve these issues very well unless we get
more specific about loss functions.  I haven't looked at the work by
Judge and Bock for several years, but I believe that it concentrates
on quadratic (in the regression parameters) loss, with the usual
minimum expected loss as the criterion.  Perhaps the results are
different when the loss function is quadratic in some measure of
predictive performance instead.  It is quite conceivable to me that
the ranking of various procedures could be reversed by choosing one or
the other risk criterion.

My main point stands, however.  An estimator is simply a function of
the data.  It is neither good nor bad, "automatic" or "thoughtful" on
its own.  It is how we use them in a given situation that counts.
--
T. Scott Thompson              email:  thompson@atlas.socsci.umn.edu
Department of Economics        phone:  (612) 625-0119
University of Minnesota        fax:    (612) 624-0209

I very much agree with Scott Thompson's thoughful post. More often
than not, the best prediction arises from a full model fit
(i.e. collinearities aren't really all that dangerous) followed
by shrinkage. Spiegelhalter (Stat in Med 5:421-33, 1986) says
this better than I could.

-- 
----------------------------------------------------------------------------
Frank E Harrell Jr			feh@biostat.mc.duke.edu
Associate Professor of Biostatistics
Division of Biometry                    Duke University Medical Center