🚀 go-pugleaf

ted@dgbt.doc.ca (Ted Grusec) writes:

[Summary of two prior postings:
  Ted wrote asking for help in finding a program to handle a 260 cell
  within-subject design.  I replied complaining that such large designs
  almost inevitably violated key assumptions and recommended fewer,
  more focussed (i.e., small, preferrably 1, df in the numerator).
  And now Ted replies:]

>Thanks. But I should point out that, in real world applications of
>experimental design, people sometimes want answers to explicit
>questions that lead you into designs that, whether you like it or not,
>result in violations of theoretical assumptions.  Also, some questions
>are pressing - important decisions need to made and there are narrow
>time limits.  All of this was true in my case.  

Important decisions in the real world are the very ones for which we
should be most concerned about theoretical assumptions.  An important
decision means that we should get the analysis right.  Unlike
between-subject analysis where the analysis is usually fairly robust
to violations, similar violations (and additional ones like sphericity
that don't even exist for between designs) in within-s designs can
lead to VERY misleading conclusions.  So, the conditions you describe
are those in which I think we should worry most about assumptions.

I realize that experimental DESIGNS often get forced on the analyst
that are not ideal.  But I wasn't talking about design issues but
rather about analysis issues.  Just because someone collected data from
a 2x10x13 design does not mean we need to analyze every possible
degree of freedom.  Rather, I was suggesting that the analyst, in
collaboration with the decision makers who will use the results of the
analysis, think about which of the 260 independent questions they
might ask of their data they really want to know about.  My
experience is that there is usually a reason why someone wanted 10
different levels for a factor.  What was that reason?  Was someone
looking for a linear or higher-order trend across those levels?  Or
were perhaps 4 of those levels alike in some way and the other 6 were
alike in some other way?  The answers to such questions can be
answered using one-df contrasts, which do not depend on any messy
and almost surely violated within-subject assumptions.  Also, such
questions can be answered using any standard multiple regression
program (as we outline in our textbook).  I was not suggesting that an
expensive experiment be redesigned and redone.  I was just suggesting
that an expensive experiment required a more thoughtful, and
probably simpler analysis, instead of just a bigger computer program.

My comments are motivated by my experience, perhaps not generalizable to
Ted's situation, of consulting with researchers, primarily cognitive
psychologists, who bring me outputs from huge within-subject analyses
that they can't understand.  I find that if I push them just a bit
they can start telling me why they included the levels that they did
and that we can then generate a list of questions THEY want to ask of
their data; these questions are often not the same ones that people
writing huge within-subject anova programs ASSUME that they want to
ask.  That is a theme throughout our textbook--to free researchers
from the tyranny of the programmers.  The unkind tone in my earlier
posting was due to my firm belief that the answer to a tough analysis
problem is almost never a bigger program.  Someone once told me of a
computer lab in an economics department that had a big sign saying,
"Think First, Regress Later."

>For many years, I taught experimental design and stat. Having long
>since left academia, I am sometimes tempted to offer a course which
>would sprinkle some "real world" constraints into design applications.
>It's something, I'm afraid, many academics are unaware of, as I too
>was unconscious of when I was in that situation.

I'm well aware of real world design constraints, both from my own
experience and from helping others make sense of designs that had been
forced on them for one reason or another.  But my point is that those
are the very situations in which a thoughtful, focussed analysis that
asks a few specific questions can be most helpful.  

gary mcclelland
univ of colorado
mcclella@yertle.colorado.edu