🚀 go-pugleaf

I've been looking at a lot of statistical papers reporting  simulation
experiments lately  and  I  must  say  that  I'm  pretty  appalled  by
statisticians' disregard of basic design and analysis principles  when
it comes to their own experiments.  For example, one reference  has  a
table of 120 empirical p-values, each based on 1000 replications;  the
nominal level is 0.05 and the authors flag as unacceptable any p-value
that is bigger than 0.06.  But the standard error for the  numbers  is
about 0.007: you're going to see 8 or 9 p-values bigger than  0.06  on
the basis of chance alone!  Questions of multiple inference and  power
were not considered at all.

Is there a good source that  discusses  the  design  and  analysis  of
statistical simulations, and especially how to compute and interpret a
relatively   large  number  of  empirical  p-values?   Something  that
includes recommendations on sample size and randomization method---ie.
whether or not to use the same random seeds for all experiments--would
be nice.

-- 
Randy Tobias          SAS Institute Inc.     sasrdt@unx.sas.com
(919) 677-8000 x7933  SAS Campus Dr.         72450.2545@compuserve.com
(919) 677-8123 (Fax)  Cary, NC   27513-2414

   Faith, faith is an island in the setting sun.
   But proof, yes: proof is the bottom line for everyone.
                                                       -- Paul Simon