🚀 go-pugleaf

In article <batra.712560431@beagle>, batra@boulder.Colorado.EDU (sajeev batra) writes:
|> Suppose I have a classifier that classifies objects into one of n 
|> classes (populations).
|> I am wondering: what are some of the better ways to score the
|> prediction accuracy of my classifier?  All classifications & 
|> misclassifications have equal cost. 
|> 
|> Here the ways that I'm aware of:
|> 
|> 1) (c1 + c2 + ... + cn)/total no. of predictions made
|>    where cn is the no. of correct predictions made in class n
|> 
|> 
|> 2) sample correlation coefficient which gives the correlation between
|>    the prediction and the actual.  the coefficient is always between
|>    -1 and +1.  There will be a coefficient for each class.  this
|>    coefficient takes into account type 1 and type 2 errors.
|> 
|> 
|> What other popular ways are there to score the prediction accuracy???  

I always like to use the following to measure the quality of classification
of a given network:

	o efficiency for class i: this tells you how many of the patterns of
class i were classified as class i elements. So we take

			(no. of i-elements classified as i-elements)
		eff(i)= -------------------------------------------
				(total no. of i-elements)

	o purity for class i: this tells you how pure your 'i-element' sample is. We define it as 

			(no. of i-elements classified as i-elements)
		pur(i)= -------------------------------------------
			(no. of elements classified as i-elements)

Frank Block

---

A quantitative assessment of the relative merits of two discriminant 
rules, d1 and d2, can be obtained by considering the "leave-one-out 
misclassification rate."  That is, given a data set X={x1,...,xn} of n 
data points drawn from two classes, we leave each data point out in turn, 
calculating the discriminant rules from the remaining n-1 observations; 
then we classify the omitted data point separately, based on each 
discriminant.  We proceed thusly n times, counting the number of cases mj 
in which the omitted point xi is classified incorrectly by discriminant
rule dj.  Our "leave-one-out misclassification rate" is then Rj(X) = mj/n.
The discriminant rule that yields the smallest Rj is quantitatively better
*under this measure, based on this data set.*

See Silverman, B.W., (1986), *Density Estimation*, Chapman&Hall, p. 124.

Along these same lines, "likelihood crossvalidation" is a technique to 
judge the adequacy of fit of a statistical model, and is related to the
Kullback-Leibler information.  While this may be more intuitive when
applied to density-based discriminants, it is more generally applicable.

Again, see Silverman, B.W., (1986), *Density Estimation*, p. 52,
and 
Stone, M., (1974), "Cross-validatory choice and assessment of statistical 
predictions," *J. Roy. Statist. Soc. B*, Vol. 36, pp. 111-147.  
and
Hall, P., (1987), "On Kullback-Leibler loss and density estimation,"
*Ann. Stat.*, Vol. 15, No. 4, pp. 1491-1519.

cep

---