🚀 go-pugleaf

As an author of a benchmark DHAMPSTONE (posted earlier on this notes file)
and a researcher in this field I should point out the following
serious misconceptions/myths about the above two benchmarks.  I hope
people stop misusing these benchmarks after they have read this.

Myth 1.  It measures the cache performance.
   
   The benchmarks did not incorporate any referencing behavior at the
   address level.  That is, they do not try to mimic the spatial or
   temporal locality in systems programs.  Even worse, because the benchmarks
   were designed to be very small, they will fit in most reasonable
   caches.  Thus running a benchmark on a cache machine in stand-alone
   mode will cause no cache misses except the initial loading.  For
   comparison purposes, runing the benchmark on two different cache
   machines is acceptable as long as it is understood that the relative
   performance is that of the CPU without the usual cache
   degradation. To put it differently, the measures show the performance
   with infinite cache.  Comparing a cache machine with a non-cache is
   also acceptable with the same understanding that the performance of the
   cache machine will be slightly over estimated.

Myth 2.  It measures the paging performance.

   Same comments as above.

Myth 3.  It measures the compiler performance.

   The benchmarks were based on high level language statistics.  No
   information was incorporated regarding  optimizing compilers and how
   they might alter the statistics.  In fact, using a good optimizing
   compiler will totally distort the performance figures.  Since the
   benchmarks are synthetic programs which do not do any useful
   computation, but merely reproduce the various measures (CALL frequency,
   parameter distribution, etc) some statements in the program are not
   logically necessary.  For example the statement x = y + z; where x is never
   used again in the program is not logically necessary and will be
   removed by a smart compiler.  However, the statement was introduced to
   produce a desired statistics on parameter and variable referencing,
   type of operation and so on.  Benchmarks were designed to evaluate
   machine performance and not compiler performance.  Some
   impact of the compiler on the performance is unavoidable.  But to
   minimize the distortions, DO NOT USE AN OPTIMIZING COMPILER when
   using the above benchmarks.

Myth 4.  It measures the absolute machine performance.

   Benchmarks provide only a relative measure of performance not an
   absolute one.  Thus if a machine X runs with a speed of 100
   Dhampstones (Dhampstone/sec) there is no simple way to predict
   the speed of your "troff" or "cc" job.
   However,  if another machine Y has a speed of 25 Dhampstones then one
   can safely assume that a reasonable mix of systems programs
   (troff, edit, compile etc) would run 4 times faster on machine X than
   on machine Y.


Janak H. Patel
University of Illinois
ARPA: uicsg!patel@uiuc.edu
UUCP: ihnp4!uiucdcs!uicsg!patel