🚀 go-pugleaf

ps26 (Paul_J_SMITH@UMAIL.UMD.EDU) wrote:
: Dear Phil -- You haven't told us enough to enable us to validate or
: invalidate your advice.  There are lots of things that can go wrong with
: logistic regression, especially with 6-8 predictors and only 34
: observations.  An a priori likely explanation of the problem is that the
: predictor variables fall into two distinct 'clouds' in the space of
: predictor variables, and that one cloud is nearly all 'successes' and the
: other is nearly all 'failures.'  If this were the case, then for a certain
: linear combination of predictors z = b1*x1 + ... +b8*x8  a graph of response
: vs. z would look like this:

:       1 |                             *  *   *** **
:         |
:         |
:         |
:       0 |     *  * *   *   * *   **
:         ------------------------------------------------> z

: If this were the case, the slope in the logistic regression of Y vs. z would
: go to infinity, and so would the intercept, while their ratio converged to a
: constant.  In terms of the original 8 predictors, some regression
: coefficients would go to +/- infinity, some might stay finite or just wander
: aimlessly, and the intercept would do who knows what.
:         Another possibility is a severe multicollinearity in the
: predictors.  This would manifest itself as aimless wandering of all iterates
: of the regression coefficients, since big changes in the b's would have
: negligible effect on the logistic likelihood.
:         In view of all this, it doesn't seem adequate to tell the student to
: 'take the money and run.'  It isn't clear whether the money is real or
: counterfeit.  What can be done is this:  look for multicollinearity among
: the predictors, and if it is present in gross amounts, reduce the predictor
: set accordingly.  Note that with lots of variables, many moderate
: correlations can indicate some complex linear dependence among the x's.  A
: principal component analysis among the x's might be in order.  A cluster
: analysis might help to pick up the 'separate clouds of points' problem if it
: is there.  Creating a new variable to represent the clustering might be
: helpful.  Of course, any new variables that you create should have
: scientific meaning.
:         Good luck to you and your student.

: Paul J. SMITH, Mathematics Department, University of Maryland at College Park
: Email:Paul_J_SMITH@umail.umd.edu (ps26)
: Phone: (301) 405-5104

--

This is a good overview of some of the possible problems. However,
there is an additional possibility, closely related to the above
observations, that sometimes crops up in in studies like
these. 

Roughly speaking, it goes like this: The researcher chooses a
regressor variable (or variables) known to have some influence on the
binary outcome and also chooses a selection of variables which s/he
wishes to evaluate for possible influence on the outcome.

The Fisher information for the problem is given by

     sum ( p_i * q_i * t(X_i) * X_i )
    over i

where i indexes the subject, p_i is the probability of response, q_i =
1- p_i, X_i is the vector of regressors for subject i, and t( ) is the
transpose.

Now, if it turns out that the researcher has choosen a regressor(s)
which predict well the outcome, then the values of p_i*q_i will all
tend to be rather small. So, there is little information about the
effects of the candidate regressors. Since the asymptotic (co)variance
is the inverse of the information matrix, one might also note that the
coefficients will have enormous variances.

There are predictable numerical consequences of this. The first is
illustrated nicely by Paul's graph. Given one good regressor and five
to seven additional candidates, it is fairly easy to find a direction
in the regressor space which results in the situation shown in the
graph. A heuristic argument to this effect says "If you have 20
successes and 14 failures (say) and one variable which allows you to
classify all but 5 successes and 3 failures with near certainty, then
when you examine six additional unrelated variables you stand a good
chance of being able to find some combination which will seem to
predict those eight cases."

Another consequence is that multicollinearity per se is not the
issue. Rather than examining sum ( t(X_i) * X_i ), one should examine
an estimate of the Fisher information. (For practical purposes, I
suppose that one would do this by estimating \hat p_i using only a few
regressors--perhaps chosen on a priori grounds--and then compute 
sum( \hat p_i * \hat q_i * t(X_i) * X_i ) including all of the candidates 
in X_i.)

Another issue (and one that I hope others can discuss) has to do with
the use of asymptotic theory to guide model selection in a
circumstance like this. Presumably, the validity of asymptotics would
depend more on sum(p_i*q_i) than on N, so if the setup includes good
predictors asymptotic theory may be worthless unless the number of
observations is truly huge.  [As an aside, I suspect that this results
in stepwise selection procedures for binary regressions leading to
nonsensical results even more frequently than their linear regression
counterparts.] Exact methods would be useful here. Of course, a simple
permutation test can be used for the hypothesis that all regression
coefficients are zero, and if some or all of the regressors are
categorical, then permutation tests tests for other hypotheses are
possible.

In addition to Paul's suggestions, I would suggest: 

Do a permutation test of the hypothesis that there are no useful
predictors (randomly permute regressor vectors against the outcomes,
compute a global index of fit--the loglikelihood is a good
candidate--record the result, repeat many times, tally the fraction of
instances in which the index of fit from the random permutations
equals or exceeds the value from the data and cite this as the
p-value.) This may seem like an uninteresting test, but at least you
can gain assurance that the regressors actually do predict the
outcome.

Look at the (p_i*q_i) using the single best (univariate) predictor to
determine \hat p_i. If all but a few of these are small, then you will
not be able to make sense of the contributions of additional
regressors. If you feel inclined to add an additional variable, remind
yourself that the sample size is (in effect roughly) equivalent to
2*sum(p_i*q_i).

You may also wish to look at matrix given by sum( \hat p_i * \hat q_i
* t(X_i) * X_i ). If some of the columns are nearly dependent, then
you won't be able to resolve the independent effects of those
columns, although it may be possible to ascertain that one or both
have an effect.

A further word about Paul's suggestion regarding cluster
analysis. What you are looking for is a linear boundary between the
successes and the failures. Thus, k-means and many other automatic
clustering programs may not be as helpful as a hand search of the
data. A useful graphical technique is based on "Andrews plots"-- I
think Edward Wegman has published on this (or will soon--he gave a talk
on this at Interface 93).

Good luck!

 ____________________________________________________________________________
|			|						     |
|Charles C. Berry	|   Department of Family and Preventive Medicine     |
|			|   University of California, San Diego		     |
|ccberry@ucsd.edu	|   La Jolla, San Diego 92093-0622		     |
|cberry@tajo.ucsd.edu	|   (619) 534-2098 (voice) (619) 534-4642 (FAX)	     |
|_______________________|____________________________________________________|