🚀 go-pugleaf

On 11 Jun 2002 13:17:22 -0700, demiurg@tau.ac.il (Alexander Sirotkin)
wrote:

 ... 
> More details:
> 10000 records, with 562 independent variables and one dependent
> variables. All 562 varriables are equally important, so that variable
> selection is meaningless, only dimension reduction may make sense 
> (but PCA does not, due to missing values problem).
> 
> The dependent variables is factor with 86 different values.
> Hence the error rate ot 0.5 is really terrible.

With 86 equally popular values, 
there would be only   1.1% correct by chance, 
so that =50%  correct'  shows enormous information, 
compared to what 'biostatisticians'  ordinarily work with.
That is tremendously 'good' -- we would say 'terrific' rather
than 'terrible'.  

So, you would not be talking about 'statistically significant'
but about the problem  of reducing particular sorts of errors.

However, if there are one or two huge categories,  50% is 
not necessarily great.  In that case, one simplified problem 
is the 2-way (or several-way)  split, where you start with the 
density of one group; or just a few.

> 
> Removing all records with missing values is out of the question -
> there are too many of them.
> 
Then you went on to say, in another post, 
that the 562 make up some sort of time series.  
(Cyclic?  identical <time> frame for all 10000 cases? )
That opens up multiple choices for modeling 
the problem, with sub-choices for modeling the Missing.

[ snip, rest, including my previous, cited post.]

-- 
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html