🚀 go-pugleaf

Forwarded note at R. Hyndman's request:

Date: Tue, 24 Dec 91 09:00:56 EST
From: hyndman@mugga.stats.mu.OZ.AU (Rob Hyndman)



COMMENTS ON THE SYSTAT--STATSOFT DEBATE

Rob J. Hyndman
Statistical Consulting Centre
The University of Melbourne,
Parkville, Victoria,  3052,
Australia.

hyndman@mugga.stats.mu.oz.au

December 1991

--------------------------------------------------------------------------


SUMMARY
-------

This report was prepared for a client, Hearne Marketing Software Pty Ltd, who
distributes statistical software in Australia.  They have asked that it be
placed on STAT-L as a service to other users of statistical software.

Some of the allegations made in the current debate between SYSTAT, Inc. and
Statsoft, Inc. are discussed.  This report considers the statistical issues
that have been raised by both parties in the debate concerning the packages
SYSTAT (from SYSTAT, Inc.) and CSS:Statistica (from Statsoft, Inc.).

A few minor bugs have been found in both packages which is to be expected in
statistical software of this scope.

However, some of the problems that have been raised regarding CSS:Statistica
are much more serious.  In particular, misleading terminology is used and some
statistical concepts seem to have been completely misunderstood and misapplied
in CSS:Statistica.

Reproduction or quotation of this report should not be made without permission
from the Statistical Consulting Centre, University of Melbourne, Australia at
the above address.

----------------------------------------------------------------------------


INTRODUCTION
------------

In October 1991, Leland Wilkinson, President of SYSTAT Inc., produced a
booklet entitled "The Truth about Statsoft and CSS:Statistica" in which he
alleged that the package CSS:Statistica from Statsoft, Inc. contained various
errors. Statsoft was quick to respond with their own booklet, "Response to
`The Truth About Statsoft'".  Then, in late November, SYSTAT produced a
further brief document, "Statsoft's Reply".

This report provides some independent comments on some of the statistical
issues involved in the allegations and counter-allegations for the benefit of
the users of both packages.  I do not comment on the other allegations in the
above documents involving false advertising and plagiarism.  Also, arguments
about the design of the packages and what statistical procedures are included
in each of them are not discussed here.

The three documents above shall be referred to as:
        SYS1  The Truth about Statsoft and CSS:Statistica    (from SYSTAT)
        CSS   Response to "The Truth About Statsoft"         (from Statsoft)
        SYS2  Statsoft's Reply.                              (from SYSTAT)

I have never been connected with either SYSTAT Inc. or Statsoft Inc. and do
not regularly use the packages SYSTAT or CSS:Statistica.  However, I have been
involved in reviews of both packages in which I considered their value for
teaching and consulting in the Department of Statistics at Melbourne
University.

Where appropriate I have used the packages MINITAB and Splus for comparison
with the results produced by SYSTAT and CSS:Statistica.


CLUSTER ANALYSIS
----------------

(SYS1, p.15; CSS, p.36--37)

SYS1 reported a genuine error in the CSS:Statistica output for cluster
analysis. The most recent version of CSS:Statistica that was available to me
(version 3.0G) was still producing the incorrect tree for the dataset
concerned.  I have not checked the CSS:Statistica routines on other data.


SPECTRAL PLOTS
--------------

(SYS1, p.16; CSS, p.19--20)

"Spectral plot" is not a standard term in the statistical literature.  For
instance, it is not mentioned in the Encyclopedia of Statistical Sciences
edited by Kotz and Johnson (1982--1989).

However, `spectral analysis' is a large branch of time series analysis and
concerns the representation of a time series as the sum of sinusoidal
components.  It is also called the `frequency domain' approach to time series
analysis.  The `spectral density' is a function which provides a measure of
the contribution of each frequency to the time series.  See Chatfield (1989,
chapters 6 & 7) for discussion of these concepts.  These ideas are frequently
applied to measurements of light, sound and other wave-based data.  In this
specialised context, the signal is the time series and is decomposed as the
sum of contributions from each frequency.

The `spectral plot' discussed by SYSTAT is a series of spectra calculated from
infrared radiation measurements from a chemical reaction.  These have been
plotted at different times and each has been smoothed using splines.  This
shows the change in the spectrum over time.

The CSS:Statistica `spectral plot' is simply a 3D scatterplot in which the
data has been projected onto various planes.  It is possible to use this plot
to show a series of spectral densities if the data are defined correctly.
However, generally the term `spectral' is inappropriate for this type of plot.
Similarly, the term `spectral planes' for the planes onto which the data are
compressed is misleading.

The CSS:Statistica manual does not give any examples or discussion of how
their `spectral plot' procedure can be used for graphing spectral density
functions or indeed, why the name `spectral plot' is used in this context.


HISTOGRAMS
----------

(SYS1, p.17; CSS, p.20)

It is standard statistical practice to draw histograms of continuous data with
no space between the columns (see, e.g., Cleveland, 1985).  The idea that such
plots introduce a `blot' effect as Statsoft claim is not generally recognised
by experts in statistical graphics.  It is not mentioned in Tufte (1983),
Cleveland (1985) or Chambers et al. (1983).  However, bar plots of categorical
data are usually plotted with gaps between the bars to emphasise the
discontinuity between categories (Kotz and Johnson, 1982--1989).  The default
histograms that CSS:Statistica produces for continuous data do have space
between the bars and therefore do not conform to standard statistical
practice.

On pairwise scatterplots, CSS:Statistica produces, by default, histograms of
each variable in the diagonal cells.  Because these histograms have spaces
between the bars, the histograms do not line up with the horizontal axes.
This makes the plots misleading and it is impossible to read off values from
the graph accurately.

SYSTAT and CSS:Statistica use different scales for their histograms on the
pairwise scatterplot.  The CSS:Statistica scale enables the shape of the
histograms to be more easily compared; the SYSTAT scale enables the counts in
each bar to be more easily compared.  It is largely a matter of opinion which
choice of scale is better.

CSS shows an example in SYSTAT in which the histograms extend beyond the
diagonal cells.  There is no need for this to ever happen and it appears to be
a bug in SYSTAT.


REGRESSION
----------

(SYS1, p.18; CSS, p.24--27)

Both SYS1 and CSS give examples of ill-conditioned regression problems where
they claim the other package has produced inaccurate results.
Ill-conditioning can occur in a variety of situations.  For example, when the
regression equation fits the data perfectly, when very large numbers are
involved, and when the coefficient of variation of a variable is very large.
The problem here is: "how should the statistics package handle
ill-conditioning?"  The correct approach is to check if the calculations can
be carried out without producing inaccurate results.  If not, then an error
message should be issued.

Statsoft claim they do "not use any `artificial flags' or `checks'".  This is
a concern.  It means that ill-conditioned problems are computed as if there
were no problems; the results may be correct or they may be inaccurate.

Consider the example given in SYS1 in which the regression equation fits the
data perfectly.  MINITAB issues an error message and refuses to do the
regression.  On the other hand,  Splus uses high precision arithmetic to
calculate the coefficients and their standard errors accurately.  (The
standard errors given by Splus are accurate to the ninth decimal place.)

CSS:Statistica compute the problem but give standard errors which are only
accurate to the third decimal place.  There are two problems here.  First, the
numerical accuracy of the regression routine in CSS:Statistica is not as
accurate as Statsoft claim.  Second, CSS:Statistica should check for
ill-conditioning and issue a warning when it cannot compute the analysis
satisfactorily.

CSS gives two examples where SYSTAT will not compute an ill-conditioned
regression problem but produces an error message instead.  In my opinion, this
is a legitimate output for such problems and far better than attempting to do
the calculations and producing inaccurate results.  However, it would be
better to give more detail about what went wrong with the calculations rather
than the general error message output by SYSTAT.

CSS also gives an example in which it is claimed SYSTAT is inaccurate.  It
seems that SYSTAT has problems displaying near-zero numbers when the underflow
option is set.  This is an unfortunate bug in SYSTAT but is not a statistical
problem with the regression routine.


CORRELATION
-----------

(SYS1, p.19; CSS, p.23--24)

Here the debates also involve accuracy with ill-conditioned problems.

It is difficult to maintain numerical accuracy when very large numbers are
used, particularly when those numbers have small variance.  Therefore, with
such problems, the highest possible precision should be used.  (An even better
approach would be to redefine the problem so it is not necessary to use such
large numbers.)

The results in SYSTAT and CSS:Statistica are now both calculated using double
precision by default and should return similar answers.  The differences
highlighted in CSS are not serious.

CSS:Statistica can display correlations (and other statistics) with 15
displayed digits.  It is important to note here the difference between the
number of displayed (and stored) digits and the number of accurate digits.
The displayed results are only as accurate as the numerical algorithms used to
obtain them.  Typically, some accuracy is lost in every numerical computation
due to rounding error or numerical approximations.  Hence, results should be
displayed using less digits than are actually stored to guard against the
errors that are introduced by the calculation.

Statsoft claim the correlations are calculated by default using double
precision. Therefore, it is highly unlikely that all 15 displayed digits will
be accurate.  (When single precision is used, the number of accurate digits
will be about 5 or less.)  So, at least for this procedure, the number of
digits that can be displayed is more than can be accurately calculated.


ANOVA
-----

(SYS1, p.20; CSS, p.27--30)

SYS1 considers a two-way experiment with each of the two factors at two
levels.  However, one of the four combinations is missing.  In this case, the
interaction between the factors cannot be estimated.  However, CSS:Statistica
not only attempts to estimate the interaction, it gives it 1 degree of freedom
and concludes it is significant.  These are very serious errors indeed.  The
interaction is not estimable and has 0 degrees of freedom. Every other
statistical package I know of rightly refuses to test such interactions.  For
example, Figure 1 shows the output from MINITAB for the analysis with no
interaction and then with interaction.

+-----------------------------------------------------------------------------+
|  MTB > glm Y = A B                                                          |
|                                                                             |
|  Analysis of Variance for Y                                                 |
|                                                                             |
|  Source     DF     Seq SS     Adj SS     Adj MS       F      P              |
|  A           1     75.600     66.667     66.667  127.27  0.000              |
|  B           1      2.333      2.333      2.333    4.45  0.073              |
|  Error       7      3.667      3.667      0.524                             |
|  Total       9     81.600                                                   |
|                                                                             |
|                                                                             |
|  MTB > glm Y = A B A*B                                                      |
|                                                                             |
|  Analysis of Variance for Y                                                 |
|                                                                             |
|  Source    Model DF   Reduced DF     Seq SS                                 |
|  A                1            1     75.600                                 |
|  B                1            1      2.333                                 |
|  A*B              1            0+     0.000                                 |
|  Error            6            7      3.667                                 |
|  Total            9            9     81.600                                 |
|                                                                             |
|  + Rank deficiency due to empty cells, unbalanced nesting or collinearity.  |
|    No storage of results or further analysis will be done.                  |
+-----------------------------------------------------------------------------+
        MINITAB output for the ANOVA problem with one missing cell.


CSS claim that CSS:Statistica uses the approach described in Milliken and
Johnson (1984).  At least for the example given in SYS1, they have incorrectly
implemented the ideas in Milliken and Johnson's book.

There are other concerning things in the document CSS which indicate a lack of
statistical understanding.  They give an example of a two-way design with a
missing cell and suggest that since the marginal means for the levels of
factor B are equal, then B should not be significant in the two way ANOVA.
This is wrong.  Tests for equality of marginal means are quite different from
testing for the significance of a factor in a two way ANOVA.  The latter test
looks at the effect of the factor B after taking into account the effect of
factor A.


NON-LINEAR ESTIMATION
---------------------

(SYS1, p.21; CSS, p.30--32)

It appears that the CSS:Statistica estimates of standard errors are inaccurate
in the non-linear estimation module.  While the formulas for calculating these
standard errors rely on an asymptotic argument, it is easy enough to calculate
them to an arbitrary degree of precision.  The algorithm used in
CSS:Statistica does not seem to be very accurate.

The counter-allegations by CSS concerning SYSTAT's non-linear estimation
procedure are based on a different problem.  Non-linear estimation is a
difficult procedure and one must be careful that it does not lead to local
minima or maxima or that the procedure does not exit before convergence has
occurred.  In the ill-conditioned example given, the SYSTAT routine has
returned a result before convergence occurred.   Unfortunately, no warning is
given and the user may mistakenly interpret the result as the correct answer.
In fact, the correct answer can be obtained in SYSTAT by continuing the
iteration.


DISCRIMINANT ANALYSIS
---------------------

(SYS1, p.22--23; CSS, p.32--34)

SYS1 points out some numerical inaccuracies in the CSS:Statistica output.
These problems have been corrected in the current version of CSS:Statistica.
However, other errors have been introduced in the tabular output.  The
cumulative percent of variance is given as one hundredth what it should be
(using version 3.0G).

The second problem raised by SYS1 concerned a plot available in
CSS:Statistica.  When assessing the success of a discriminant analysis, it is
standard practice to examine scatterplots of the scores from the first two
discriminant functions.  This is a visual display of the success of the
discrimination between groups.  By definition, the first two discriminant
functions are orthogonal and so have zero correlation.  The regression line
plotted by CSS:Statistica on their version of this graph is inappropriate and
has no real meaning.  The associated confidence intervals are also totally
inappropriate.  There is no random variation in the slope of this line; it is
zero by definition.

As in multiple regression, the canonical coefficients that are used in each
discriminant function may be calculated as either `raw' or `standardised'
depending on whether the associated predictor variable has been standardised.
However, the resulting discriminant scores calculated from the discriminant
function with the raw coefficients and predictor variables are the same (or
differ by a constant) from the scores calculated using the standardised
coefficients and predictor variables.  The scores from each discriminant
function are by definition `standardised' to have variance one (Johnson and
Wichern, 1982, p.506; Tabachnick and Fidell, 1989, p.515).

Hence, the plot of `standardised' canonical scores offered by CSS:Statistica
is incorrect. It should not differ (except perhaps by a shift in location)
from the plot of raw canonical scores.  As noted above, a regression line on
this type of graph is inappropriate.  If such a line were plotted, it should
have zero slope.  However, CSS:Statistica produces a line that typically has
non-zero slope.

There is no standard terminology used in discriminant analysis.  Several terms
are applied to the same objects.  Hence, `discriminant functions' are also
known as `canonical variates' and `factors'.  However, the term `roots' used
by CSS:Statistica for discriminant functions is not one I have come across
before in this context in the statistical literature.


GOODNESS-OF-FIT TESTS
---------------------

(SYS1, p.24; CSS, p.34--36)

There are two different sorts of tests discussed here.  The first is a test
that the data follow a specific distributional model with specified
parameters.  The second is a test that the data follow a distributional model
with parameters estimated from the data.  In the case under discussion, the
model is a normal distribution.

The test statistics for the two types of test have different distributions and
should not be confused.  The significance points for the test with unknown
parameters are much smaller than those for the test in which the distribution
is completely specified.  Using the latter values for tests of distributions
with unknown parameters "will lead to serious errors in the recorded
significance level of the test." (Kotz and Johnson, Kolmogorov--Smirnov--Type
Tests of Fit, p.400.)

These comments apply to other goodness-of-fit tests also.

CSS:Statistica use the wrong significance levels in some of their
goodness-of-fit tests.  CSS claim that other packages (SPSS) do the same tests
which is true, but only when the distribution is specified by the user, not
estimated from the data.

SYS1 (p.24) points out other problems in the construction of the graphs, which
are valid objections.


REFERENCES
----------

Chambers, J.M., Cleveland, W.S., Kleiner, B. and Tukey, P.A., (1983) Graphical
methods for data analysis, Wadsworth: California.

Chatfield, C., (1989)  The analysis of time series: an introduction, 4th
edition, Chapman and Hall: London.

Cleveland, W.S., (1985)  The elements of graphing data, Wadsworth: California.

Johnson, R.A. and Wichern, D.W., (1982)  Applied multivariate statistical
analysis, Prentice--Hall: New Jersey.

Kotz, S. and Johnson, N.L., (1982--1989)  Encyclopedia of statistical
sciences, Wiley: New York.

Milliken, G.A. and Johnson, D.E., (1984)  Analysis of messy data. Volume 1:
designed experiments, Van Nostrand Reinhold: New York.

Tabachnick, B.G. and Fidell, L.S., (1989)  Using multivariate statistics,
Harper and Row: New York.

Tufte, E.R., (1983)  The visual display of quantitative information, Graphic
Press: Connecticut.