🚀 go-pugleaf

I managed to reduce the error rate by reducing the "minimal node deviance"
parameter of the partitioning algorithm. But it seems to me that this method 
is wrong, that it just makes a bigger tree (with more terminal nodes) which 
just memorizes the whole input.
But on the other hand, S-plus does use a different test set, so that there 
can not be generalization problems...

Very confuzing...

demiurg@tau.ac.il (Alexander Sirotkin) wrote in message news:<adrouf$h81$1@mulga.cs.mu.OZ.AU>...
> Question : How to improve (reduce) decision tree 
> classification model error rate?
> 
> I'm trying to build some decision tree classification
> model (using S-plus) and keep getting extremely bad
> results (error rate around 0.5). 
> 
> The data : 562 real predictor variables and one 
> independent variable, factor (with 86 types).
> 
> Quite a lot of missing values...
> 
> Any ideas what should I try (data preprocessing, etc...)
> to reduce error rate ?
>

[ comp.ai is moderated.  To submit, just post and be patient, or if ]
[ that fails mail your article to <comp-ai@moderators.isc.org>, and ]
[ ask your news administrator to fix the problems with your system. ]

Groups trimmed to sci.stat.math, aus.stats.s.

 - the question lacks some basic parameters which 
would limit the options -

On 8 Jun 2002 12:08:15 +1000, demiurg@tau.ac.il (Alexander Sirotkin)
wrote:

> Question : How to improve (reduce) decision tree 
> classification model error rate?
> 
> I'm trying to build some decision tree classification
> model (using S-plus) and keep getting extremely bad
> results (error rate around 0.5). 

That's bad if there are just two choices, with equal Ns.
Just how bad is it?
 
> The data : 562 real predictor variables and one 
> independent variable, factor (with 86 types).

85 dummy variables?  Or, ordered in a sensible way to
be 3 or 4 dummies, or 15?

> 
> Quite a lot of missing values...

Scattered randomly across 600 variables?
across 500?  5000?  5 million cases?
What do you lose if you drop all Subjects
with missing? - all variables with missing?
The numerically comprehensive way to try to 
deal with missing is to create a dummy variable 
for Missing/Not .  For a real Prediction equation,
for the future, you seldom see that.  It is not
going to be worth it except for important variables
that *might* , in real life, end up Missing.

> Any ideas what should I try (data preprocessing, etc...)
> to reduce error rate ?

I like to create composite scores that are meaningful.
A reference that was posted recently suggests that 
variable selection might be best achieved by generating
equations in relatively small sub-samples, so that the
validation can magnify the error rate.  That strikes me
as an intriguing suggestion -- I have always been very
skeptical about stepwise variables selection.

There's hardly a clue about what makes up these data, 
so my 'ideas'  are limited.  

> [ comp.ai is moderated.  
 < snipped >

-- 
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html

Hi.

Thanks for the response.

More details:
10000 records, with 562 independent variables and one dependent
variables. All 562 varriables are equally important, so that variable
selection
is meaningless, only dimension reduction may make sense (but PCA does
not,
due to missing values problem).

The dependent variables is factor with 86 different values.
Hence the error rate ot 0.5 is really terrible.

Removing all records with missing values is out of the question -
there are too
many of them.

BTW, using spepwise method (if I get the term right - dividing the
sample to say 4 subsamples
and build a classification model for each one), does help...




Rich Ulrich <wpilib@pitt.edu> wrote in message news:<mka9gukh228jdm993ht1erdo793j6f52ig@4ax.com>...
> Groups trimmed to sci.stat.math, aus.stats.s.
> 
>  - the question lacks some basic parameters which 
> would limit the options -
> 
> On 8 Jun 2002 12:08:15 +1000, demiurg@tau.ac.il (Alexander Sirotkin)
> wrote:
> 
> > Question : How to improve (reduce) decision tree 
> > classification model error rate?
> > 
> > I'm trying to build some decision tree classification
> > model (using S-plus) and keep getting extremely bad
> > results (error rate around 0.5). 
> 
> That's bad if there are just two choices, with equal Ns.
> Just how bad is it?
>  
> > The data : 562 real predictor variables and one 
> > independent variable, factor (with 86 types).
> 
> 85 dummy variables?  Or, ordered in a sensible way to
> be 3 or 4 dummies, or 15?
> 
> > 
> > Quite a lot of missing values...
> 
> Scattered randomly across 600 variables?
> across 500?  5000?  5 million cases?
> What do you lose if you drop all Subjects
> with missing? - all variables with missing?
> The numerically comprehensive way to try to 
> deal with missing is to create a dummy variable 
> for Missing/Not .  For a real Prediction equation,
> for the future, you seldom see that.  It is not
> going to be worth it except for important variables
> that *might* , in real life, end up Missing.
> 
> > Any ideas what should I try (data preprocessing, etc...)
> > to reduce error rate ?
> 
> I like to create composite scores that are meaningful.
> A reference that was posted recently suggests that 
> variable selection might be best achieved by generating
> equations in relatively small sub-samples, so that the
> validation can magnify the error rate.  That strikes me
> as an intriguing suggestion -- I have always been very
> skeptical about stepwise variables selection.
> 
> There's hardly a clue about what makes up these data, 
> so my 'ideas'  are limited.  
> 
> > [ comp.ai is moderated.  
>  < snipped >

Additional piece of information : 

There is an information in this data set that may current
classification model does not use. The model does not take into 
account taht these 562 real predictor variables (not records !!!)  
have order, meaning that the variables are not just x[1], x[2],..., x[561]
but it is important that x[3] comes after x[2] (because these variables
are actually samples from continues function).

what I'm trying to do, actually, is to build a classification model
using functions (sampled at 562 points) as independent variables !


Rich Ulrich <wpilib@pitt.edu> wrote in message news:<mka9gukh228jdm993ht1erdo793j6f52ig@4ax.com>...
> Groups trimmed to sci.stat.math, aus.stats.s.
> 
>  - the question lacks some basic parameters which 
> would limit the options -
> 
> On 8 Jun 2002 12:08:15 +1000, demiurg@tau.ac.il (Alexander Sirotkin)
> wrote:
> 
> > Question : How to improve (reduce) decision tree 
> > classification model error rate?
> > 
> > I'm trying to build some decision tree classification
> > model (using S-plus) and keep getting extremely bad
> > results (error rate around 0.5). 
> 
> That's bad if there are just two choices, with equal Ns.
> Just how bad is it?
>  
> > The data : 562 real predictor variables and one 
> > independent variable, factor (with 86 types).
> 
> 85 dummy variables?  Or, ordered in a sensible way to
> be 3 or 4 dummies, or 15?
> 
> > 
> > Quite a lot of missing values...
> 
> Scattered randomly across 600 variables?
> across 500?  5000?  5 million cases?
> What do you lose if you drop all Subjects
> with missing? - all variables with missing?
> The numerically comprehensive way to try to 
> deal with missing is to create a dummy variable 
> for Missing/Not .  For a real Prediction equation,
> for the future, you seldom see that.  It is not
> going to be worth it except for important variables
> that *might* , in real life, end up Missing.
> 
> > Any ideas what should I try (data preprocessing, etc...)
> > to reduce error rate ?
> 
> I like to create composite scores that are meaningful.
> A reference that was posted recently suggests that 
> variable selection might be best achieved by generating
> equations in relatively small sub-samples, so that the
> validation can magnify the error rate.  That strikes me
> as an intriguing suggestion -- I have always been very
> skeptical about stepwise variables selection.
> 
> There's hardly a clue about what makes up these data, 
> so my 'ideas'  are limited.  
> 
> > [ comp.ai is moderated.  
>  < snipped >

In sci.stat.math Alexander Sirotkin <demiurg@tau.ac.il> wrote:
> There is an information in this data set that may current
> classification model does not use. The model does not take into 
> account taht these 562 real predictor variables (not records !!!)  
> have order, meaning that the variables are not just x[1], x[2],..., x[561]
> but it is important that x[3] comes after x[2] (because these variables
> are actually samples from continues function).

> what I'm trying to do, actually, is to build a classification model
> using functions (sampled at 562 points) as independent variables !

Ah, you're saying that your independent variables are actually the values 
of a time series.  In that case, some of your difficulty may be coming 
from the fact that they're so highly correlated that it's difficult to 
distinguish the weights that should be assigned them.

It looks like you're really trying to estimate the coefficients of a 
finite-impulse-response filter whose output is then somehow quantized into 
80-something not-necessarily-equal-sized steps.  I'm not really sure how 
to deal with such a problem.  One thing that's clear is that anything 
resembling classical discriminant analysis isn't going to work with your 
sample size of around 10,000, because it would require estimating around 
45,000 parameters which is almost 5 times the number of independent 
observations you have.  I also have a very strong hunch that your apparent 
"success" with stepwise methods is actually success at fitting the noise, 
rather than signal, in your data.

It seems that you want to reduce each time series to a a few parameters and
cluster those parameters.

The Classification Society of North America page
http://www.pitt.edu/~csna/
might be of help.   I have seen this kind of problem at past CSNA meetings.
Browse the abstracts etc.

You might post a message on <class-l> describing your data and the kind of
questions you have.


Eric Bohlman wrote:

> In sci.stat.math Alexander Sirotkin <demiurg@tau.ac.il> wrote:
> > There is an information in this data set that may current
> > classification model does not use. The model does not take into
> > account taht these 562 real predictor variables (not records !!!)
> > have order, meaning that the variables are not just x[1], x[2],..., x[561]
> > but it is important that x[3] comes after x[2] (because these variables
> > are actually samples from continues function).
>
> > what I'm trying to do, actually, is to build a classification model
> > using functions (sampled at 562 points) as independent variables !
>
> Ah, you're saying that your independent variables are actually the values
> of a time series.  In that case, some of your difficulty may be coming
> from the fact that they're so highly correlated that it's difficult to
> distinguish the weights that should be assigned them.
>
> It looks like you're really trying to estimate the coefficients of a
> finite-impulse-response filter whose output is then somehow quantized into
> 80-something not-necessarily-equal-sized steps.  I'm not really sure how
> to deal with such a problem.  One thing that's clear is that anything
> resembling classical discriminant analysis isn't going to work with your
> sample size of around 10,000, because it would require estimating around
> 45,000 parameters which is almost 5 times the number of independent
> observations you have.  I also have a very strong hunch that your apparent
> "success" with stepwise methods is actually success at fitting the noise,
> rather than signal, in your data.

how about independent components analysis---
which allows for nonlinear relationships---
just wondering....
"Arthur J. Kendall" <Arthur.Kendall@verizon.net> wrote in message
news:3D0748DB.E857635D@verizon.net...
> It seems that you want to reduce each time series to a a few parameters
and
> cluster those parameters.
>
> The Classification Society of North America page
> http://www.pitt.edu/~csna/
> might be of help.   I have seen this kind of problem at past CSNA
meetings.
> Browse the abstracts etc.
>
> You might post a message on <class-l> describing your data and the kind of
> questions you have.
>
>
> Eric Bohlman wrote:
>
> > In sci.stat.math Alexander Sirotkin <demiurg@tau.ac.il> wrote:
> > > There is an information in this data set that may current
> > > classification model does not use. The model does not take into
> > > account taht these 562 real predictor variables (not records !!!)
> > > have order, meaning that the variables are not just x[1], x[2],...,
x[561]
> > > but it is important that x[3] comes after x[2] (because these
variables
> > > are actually samples from continues function).
> >
> > > what I'm trying to do, actually, is to build a classification model
> > > using functions (sampled at 562 points) as independent variables !
> >
> > Ah, you're saying that your independent variables are actually the
values
> > of a time series.  In that case, some of your difficulty may be coming
> > from the fact that they're so highly correlated that it's difficult to
> > distinguish the weights that should be assigned them.
> >
> > It looks like you're really trying to estimate the coefficients of a
> > finite-impulse-response filter whose output is then somehow quantized
into
> > 80-something not-necessarily-equal-sized steps.  I'm not really sure how
> > to deal with such a problem.  One thing that's clear is that anything
> > resembling classical discriminant analysis isn't going to work with your
> > sample size of around 10,000, because it would require estimating around
> > 45,000 parameters which is almost 5 times the number of independent
> > observations you have.  I also have a very strong hunch that your
apparent
> > "success" with stepwise methods is actually success at fitting the
noise,
> > rather than signal, in your data.
>

On 11 Jun 2002 13:17:22 -0700, demiurg@tau.ac.il (Alexander Sirotkin)
wrote:

 ... 
> More details:
> 10000 records, with 562 independent variables and one dependent
> variables. All 562 varriables are equally important, so that variable
> selection is meaningless, only dimension reduction may make sense 
> (but PCA does not, due to missing values problem).
> 
> The dependent variables is factor with 86 different values.
> Hence the error rate ot 0.5 is really terrible.

With 86 equally popular values, 
there would be only   1.1% correct by chance, 
so that =50%  correct'  shows enormous information, 
compared to what 'biostatisticians'  ordinarily work with.
That is tremendously 'good' -- we would say 'terrific' rather
than 'terrible'.  

So, you would not be talking about 'statistically significant'
but about the problem  of reducing particular sorts of errors.

However, if there are one or two huge categories,  50% is 
not necessarily great.  In that case, one simplified problem 
is the 2-way (or several-way)  split, where you start with the 
density of one group; or just a few.

> 
> Removing all records with missing values is out of the question -
> there are too many of them.
> 
Then you went on to say, in another post, 
that the 562 make up some sort of time series.  
(Cyclic?  identical <time> frame for all 10000 cases? )
That opens up multiple choices for modeling 
the problem, with sub-choices for modeling the Missing.

[ snip, rest, including my previous, cited post.]

-- 
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html

Have you considered CART? A friend of mine swears by it:

  http://www.google.com/search?q=classification+and+regression+trees+cart

Bjarne

In comp.ai Alexander Sirotkin <demiurg@tau.ac.il> wrote:
: I managed to reduce the error rate by reducing the "minimal node deviance"
: parameter of the partitioning algorithm. But it seems to me that this method 
: is wrong, that it just makes a bigger tree (with more terminal nodes) which 
: just memorizes the whole input.
: But on the other hand, S-plus does use a different test set, so that there 
: can not be generalization problems...

: Very confuzing...

: demiurg@tau.ac.il (Alexander Sirotkin) wrote in message news:<adrouf$h81$1@mulga.cs.mu.OZ.AU>...
:> Question : How to improve (reduce) decision tree 
:> classification model error rate?
:> 
:> I'm trying to build some decision tree classification
:> model (using S-plus) and keep getting extremely bad
:> results (error rate around 0.5). 
:> 
:> The data : 562 real predictor variables and one 
:> independent variable, factor (with 86 types).
:> 
:> Quite a lot of missing values...
:> 
:> Any ideas what should I try (data preprocessing, etc...)
:> to reduce error rate ?
:>

[ comp.ai is moderated.  To submit, just post and be patient, or if ]
[ that fails mail your article to <comp-ai@moderators.isc.org>, and ]
[ ask your news administrator to fix the problems with your system. ]