🚀 go-pugleaf

This is a repost to correct a typo . . .

In article <1992Oct29.223353.887@access.usask.ca>, bickis@skmath3 (M. Bickis) writes:
[and I edited . . .]
> I was analyzing a large but somewhat sparse contingency
> table on BMDP 4F.  I noticed, almost by accident, that the program was
> not calculating the degrees-of-freedom for some log-linear models correctly.

> A bit of playing around convinced me that the problem was due to the
> sparseness of the table. 
> Finally, I came upon an almost parenthetical comment in the
> 1990 manual p.304:
> "We recommend that you add a constant to all cells when you have empty
> cells in your data that are not to be treated as structural zeros. 
> Otherwise, some of the analyses ARE NOT PERFORMED CORRECTLY."
> 
> I have just told my class that BMDP has one of the best log-linear
> programs around.  Now I wonder if that was bad advice.

Let's be fair.  This is not an easy problem.  The correct number of degrees of
freedom depends not only on the pattern of structural zeros, but on the model
being fitted to them.  For details, see Savage IR (1973), "Incomplete
Contingency Tables:  Conditions for the Existence of Unique MLE" in Jagars P
and Rade L eds. Mathematics and Statistics:  Essays in Honour of Harald 
Bergstrom.  Goteborg, Sweden: (don't have the publisher. Sorry.)

For those who have not yet been exposed to the problem, consider fitting
a quasi-independence model to an r*c table, i.e., a model that specifies
row/column independence over a subset of the table (example, type of crime
against campus.  If one of the crimes were auto theft, we would not expect 
a homogeneity of proportions model to extend to those campuses where no cars 
were allowed. For those campuses, structural 0's would be fitted to the 
auto-theft cells and we might asks whether a model of homogeneity of 
proportions fit the rest of the table.)

If only a few cells contain structural zeros, subtract 1 degree of freedom for
each structural 0.  This will be incorrect if an entire row, say, contains
structural 0's because the rule of thumb gives (r-1)*(c-1)-c, whereas the 
correct number of degrees of freedom is (r-2)*(c-1).

But it's just not as simple as checking for empty rows and columns.  Consider
the 4 * 4 table (3*3=9 df) with structural zeros indicated by 0

     . . 0 0 
     . . 0 0
     0 0 . .
     0 0 . .

The rule of thumb says that there is 1 df (9-8=1) but clearly there are 2--
one for each of the 2*2 tables.  With multi-dimensional tables the problem gets
harder because, as I already mentioned, the correct number of df depends on not
just the pattern of structural zeros, but also on the model being fitted to 
them.

Perhaps I'm being too pesimistic.  It may take just one determined person to 
look at Savage's results and turn them into usable code.  It would make a nice
Applied Statistics algorithm if it could be written to be compatible with 
Shelby Haberman's programs for analyzing multidimensional tables, also
published in Applied Statistics.

In article <1992Oct30.120219.480@hnrc.tufts.edu> jerry@hnrc.tufts.edu 
(Jerry Dallal) writes in reply to me:
>> I was analyzing a large but somewhat sparse contingency
>> table on BMDP 4F.  I noticed, almost by accident, that the program was
>> not calculating the degrees-of-freedom for some log-linear models correctly.
>>                           [ ... ]
>> "We recommend that you add a constant to all cells when you have empty
>> cells in your data that are not to be treated as structural zeros. 
>> Otherwise, some of the analyses ARE NOT PERFORMED CORRECTLY."
>
>Let's be fair.  This is not an easy problem.  The correct number of degrees of
>freedom depends not only on the pattern of structural zeros, but on the model
>being fitted to them.  

I don't dispute that finding degrees of freedom for a complicated model with 
structural zeros is non-trivial.  My point was that in this table I was *not*
assuming any structural zeros.  For a complete table, the df depend *only* on
the model and not on any pattern of *observed* zeros.  BMDP 4F has a
feature for identifying structural zeros which I had not invoked.  If
the program was arbitrarily assuming that observed zeros were
structural, at the very least it should have printed out a message to
that effect.



>>>>>>>>>>>>>>>>>>>bickis@math.usask.ca<<<<<<<<<<<<<<<<<<<<<<<<<<<
Mik Bickis       \,,,       .*#*.  \    University of Saskatchewan
Department of        \,,,  ^\\|//^  ```\   Saskatoon, Saskatchewan
Mathematics & Statistics \   )|(        ```\   Canada      S7N 0W0
<<<<<<<<<<<<<<<<<<<S|A|S|K|A|T|C|H|E|W|A|N|>>>>>>>>>>>>>>>>>>>>>>>

In article <1992Oct30.135542.482@hnrc.tufts.edu>, jerry@hnrc.tufts.edu (Jerry Dallal) writes:

|> For those who have not yet been exposed to the problem, consider fitting
|> a quasi-independence model to an r*c table, i.e., a model that specifies
|> row/column independence over a subset of the table (example, type of crime
|> against campus.  If one of the crimes were auto theft, we would not expect 
|> a homogeneity of proportions model to extend to those campuses where no cars 
|> were allowed. For those campuses, structural 0's would be fitted to the 
|> auto-theft cells and we might asks whether a model of homogeneity of 
|> proportions fit the rest of the table.)

I have a related (proably naive) question:

I am doing spatial trend removal with a median polish. A grid is overlayed
on a 2-dimensional data set. Points are first linked to their nearest grid 
point (some grid points may have more than one point, others may have 
none--0's).  Then, medians are systematically removed from the rows and 
then columns of grid points. This is repeated until, within some tolerance 
level, the polish converges.

The reference that I am using suggests that, before I used the residuals
for anything, I should test for any "significant interation" (I think that's
the term he used). Can someone expound upon this (in private email if you
feel this is too trivial for sci.math.stat)? How do I test for it? What 
happens if I conclude that there is interaction (if this can be answered
generally---I assume that it depends on what I'm using the residuals for)?

The work that I allude to is appended.
-- 
James Darrell McCauley                Dept of Ag Engineering, Purdue Univ
internet: mccauley@ecn.purdue.edu     West Lafayette, Indiana 47907-1146, USA
bitnet: mccauley%ecn@purccvm          UUCP: pur-ee!mccauley

@Book{ cressie91,
  author =       "Noel A. C. Cressie",
  title =        "Statistics for Spatial Data",
  publisher =    "John Wiley \& Sons",
  year =         "1991",
  series =       "Wiley Series in Probability and Mathematical Statistics",
  address =      "New York, NY"
}

In article <1992Oct30.205117.22832@access.usask.ca>, bickis@skmath3 (M. Bickis) writes:
> I don't dispute that finding degrees of freedom for a complicated model with 
> structural zeros is non-trivial.  My point was that in this table I was *not*
> assuming any structural zeros.  For a complete table, the df depend *only* on
> the model and not on any pattern of *observed* zeros.  BMDP 4F has a
> feature for identifying structural zeros which I had not invoked.  If
> the program was arbitrarily assuming that observed zeros were
> structural, at the very least it should have printed out a message to
> that effect.
> 

Now you've piqued by curiosity.  Would you send the job to me by e-mail,
eliminating any labels/information that might be sensitive?

[I tried sending this request through REPLY rather than FOLLOWUP, but
my system is choking on your address.]