🚀 go-pugleaf

Benchmark comparisons give valid results only if the same program is presented
to each machine.  The compiler is considered part of the "machine" and its
performance contributes to the overall performance of the machine.  This study
did not present the same program to each machine.  Instead "[they] had the
same person modify or write all the tests so [they] could be sure that the same
algorithms would be used for all the processors" (page 56 of the IEEE article).
Thus the benchmark results reflect not only the individual processors' ability
to execute instructions but also the cleverness of this programmer in using
each microprocessor's instruction set and architecture.  The results reported
should not be considered true measures of the relative performance of these
microprocessors.
   Unfortunately benchmark comparisons often suffer from commercial hype and
unscientific methods.  Let the buyer beware!

                                   Alan Wiemann
                                   Aitchpeelabs
                                   Palowaltocalifornya
                                   hplabs!wiemann

The table below is a reorganization of the following table.

				Relative Performance

processor	80286	80386	68000	68020	68020	32032	32100	32100
cache (MHz)	(10)	(16)	(8)	N (16)	C (16)	(10)	N (18)	C (18)

string search	1.37	1.00	3.85	2.25	1.08	3.51	4.71	1.89
bit manipulate	5.52	2.09	5.91	2.25	1.00	5.29	3.58	1.74
linked list	3.08	1.70	4.11	1.79	1.00	2.90	2.36	1.28
quicksort	4.07	2.49	4.39	2.05	1.00	3.12	3.12	1.32
matrix trans	6.42	2.05	5.49	1.58	1.00	4.33	3.04	1.47

average		4.09	1.87	4.75	1.98	1.02	3.83	3.36	1.54

In article <322@oblio.UUCP> paf@oblio.UUCP (Paul Fronberg) writes:
)In the August issue of IEEE Micro there is a very interesting article
)concerning benchmarking 32-bit microprocessors. The following table is
)abstracted from page 57. Numbers are time in seconds. (N=no cache enabled;
)C=cache enabled). This table reflects the results for dynamic memory.
)
)	 MHz	  E	  F	 H	  I	  K
)80286	(10)	 4.89	13.63	6.59	11.20	19.39
)80386	(16)	 3.57	 5.16	3.63	 6.86	 6.20
)68000	(8)	13.73	14.61	8.79	12.08	16.59
)68020 N	(16)	 8.02	 5.55	3.84	 5.65	 4.78
)68020 C (16)	 3.84	 2.47	2.14	 2.75	 3.02
)32032	(10)	12.52	13.07	6.21	 8.57	13.07
)32100 N (18)	16.81	 8.84	5.05	 8.57	 9.17
)32100 C (18)	 6.75	 4.29	2.74	 3.63	 4.45
)
)Benchmarks are EDN 16 benchmarks modified for 32 bits. The benchmarks were
)coded in assembly code for each processor.
)
)The following EDN programs were used.
)
)Test E is a character-string search routine.
)
)Test F is a bit test, set, and reset routine.
)
)Test H is a linked-list insertion routine.
)
)Test I is a quicksort routine.
)
)Test K is a bit-matrix transposition routine.


-- 
  Lawrence Crowl		716-275-5766	University of Rochester
			crowl@rochester.arpa	Computer Science Department
 ...!{allegra,decvax,seismo}!rochester!crowl	Rochester, New York,  14627

In article <20954@rochester.ARPA>, crowl@rochester.ARPA (Lawrence Crowl) writes:
> The table below is a reorganization of the following table. [I ommitted
> this table in this posting. It contained the times for the benchmarks in
> seconds. RN]
> 				Relative Performance
> 
> processor	80286	80386	68000	68020	68020	32032	32100	32100
> cache (MHz)	(10)	(16)	(8)	N (16)	C (16)	(10)	N (18)	C (18)
> 
> string search	1.37	1.00	3.85	2.25	1.08	3.51	4.71	1.89
> bit manipulate5.52	2.09	5.91	2.25	1.00	5.29	3.58	1.74
> linked list	3.08	1.70	4.11	1.79	1.00	2.90	2.36	1.28
> quicksort	4.07	2.49	4.39	2.05	1.00	3.12	3.12	1.32
> matrix trans	6.42	2.05	5.49	1.58	1.00	4.33	3.04	1.47
> 
> average	4.09	1.87	4.75	1.98	1.02	3.83	3.36	1.54

The averaging in this table is done incorrectly. As noted in a recent CACM
article, normalized benchmark results should be averaged with a geometric
mean, not an arithmetic mean. The geometic mean of N numbers is the Nth 
root of their product. This method gives the correct results:

  RIGHT average	3.60	1.79	4.68	1.97	1.02	3.74	3.28	1.52

In this case, it doesn't seem to make all that much difference in the
conclusions. In can though, consider the following example:

		Machine A	Machine B

  Benchmark 1:	10 seconds	5 seconds
  Benchmark 2:	10 seconds	20 seconds

Look at the results of normalizing these figures to Machine A and then
taking the arithmetic mean of the results:

  		Machine A	Machine B
  Benchmark 1:	  1.0		  0.5
  Benchmark 2:	  1.0		  2.0

  arith. mean:	  1.0		  1.25

Machine B is thus 25% slower than machine A, right? Wrong. Look at what
happens when you take the *same* benchmark results, normalize to machine
B, and take the arithmetic mean:

  		Machine A	Machine B
  Benchmark 1:	  2.0		  1.0
  Benchmark 2:	  0.5		  1.0

  arith. mean:	  1.25		  1.0

Now machine B comes out looking faster! 

If you take the geometric mean, however, machine A and machine B look
equally fast regardless of how you normalize the results. 

    Radford Neal
    The University of Calgary

> 
> The averaging in this table is done incorrectly. As noted in a recent CACM
> article, normalized benchmark results should be averaged with a geometric
> mean, not an arithmetic mean.
>     Radford Neal
>     The University of Calgary

There is no clear cut evidence that the geometric mean is any more correct
than any other [Re: the Flemming and Wallace paper].  Jack Wolton of
Los Alamost published a paper in 1984 (IEEE Compcon, with the title:
Bottleneckology: something, my proceedings are lent out at the moment).
This paper touted the HARMONIC mean as the "correct" mean.  This draws
both suspect.  The proof offered by W&F is not a sufficiently
rigorous proof.  And I think a poor proof is worse than no proof.

The science and art of measurement are independent of statistics (I
realize an over simplification).  We resort to statistics because our
measurement tool on the computer are so poor.  What sense is it to
average numbers intended to show some sort of peak performance?
(Don't say average peak performance.)

I suggest reading two texts:

%A Darrell Huff
%T How to Lie with Statistics
%I Norton
%C NY
%D 1954

%A Edward R. Tufte
%T The Visual Display of Quatitative Information
%I Graphics Press
%C Cheshire, CO
%D 1983

Separate note on performance measurement:
I have been talking to colleagues at Berkeley, LBL, LLNL,
and other locations.  We will be having an informal dinner meeting,
probably to be held at UC Berkeley (to the shagrin of the Stanford people)
sometime during the end of October.  I am trying to think of a name to
characterize this group: New Generation Performance Measurement Group
(naw, make it a Ring), Bay Area Performance Measurement Ring,
Those People Interested in Improving Performance Measurement People.
Anyway, if you are in the Bay Area and are seriously interested,
contact me.  We have several mail correspondents like Jack Dongarra
and Ken Dymond, but we want to have this meeting, too.

Added reference:

%A Philip J. Flemming
%A John J. Wallace
%T How Not to Lie with Statistics: The Correct Way to Summarize
Benchmark Results
%J CACM
%V 29
%N 3
%D March 1986
%P 218-221
%X Waste of a good title.

From the Rock of Ages Home for Retired Hackers:
--eugene miya
  NASA Ames Research Center
  com'on do you trust Reply commands with all these different mailers?
  {hplabs,ihnp4,dual,hao,decwrl,tektronix,allegra}!ames!aurora!eugene
  eugene@ames-aurora.ARPA

..> `Averaging' benchmarks using the arithmetic or geometric mean

Whatever you do, you shouldn't combine benchmarks without weight factors
proportional to the importance of the jobs characterized by the benchmark
to you. You can use any weights in any type of characteristic you want,
but not all are meaningful. For example, if throughput is what you want 
to optimize, and your weights are the percentage of your job mix that is
characterized by each benchmark, then a linearly weighted arithmetic mean
is the average to use.

That, of course, assumes that there are no interactions between jobs.

Andy "Krazy" Glew. Gould CSD-Urbana.    USEnet:  ihnp4!uiucdcs!ccvaxa!aglew
1101 E. University, Urbana, IL 61801    ARPAnet: aglew@gswd-vms

In article <ames.1675> eugene@ames.UUCP (Eugene Miya) writes:
>> 
>> The averaging in this table is done incorrectly. As noted in a recent CACM
>> article, normalized benchmark results should be averaged with a geometric
>> mean, not an arithmetic mean.
>>     Radford Neal
>>     The University of Calgary
>
>There is no clear cut evidence that the geometric mean is any more correct
>than any other [Re: the Flemming and Wallace paper].  Jack Wolton of
>Los Alamos published a paper in 1984...
>This paper touted the HARMONIC mean as the "correct" mean....

Note that if the values being averaged don't have too much spread, all these
means are about the same;  also, even if the distributions of values have 
large spread, I believe the distributions being compared have to be rather 
different in form for the different methods of calculating the mean
to give different rank-orders.  (But I seem to have missed the original
article to which this refers, so I'm not sure what's being compared!)

Peter S. Shenkin	 Columbia Univ. Biology Dept., NY, NY  10027
{philabs,rna}!cubsvax!peters		cubsvax!peters@columbia.ARPA

In article <1675@ames.UUCP>, eugene@ames.UUCP (Eugene Miya) writes:

> > The averaging in this table is done incorrectly. As noted in a recent CACM
> > article, normalized benchmark results should be averaged with a geometric
> > mean, not an arithmetic mean.
> 
> There is no clear cut evidence that the geometric mean is any more correct
> than any other [Re: the Flemming and Wallace paper].  Jack Wolton of
> Los Alamost published a paper in 1984 (IEEE Compcon) [touting]
> HARMONIC mean as the "correct" mean...
> The proof offered by W&F is not a sufficiently
> rigorous proof.  And I think a poor proof is worse than no proof.

Could you elaborate on what's wrong with the proof offered by W&F? I 
re-read the article last night, and they seem to me to prove what they
set out to prove. The harmonic mean certainly doesn't work for normalized
numbers.

W&F do not claim that the geometric mean is the only good way to average
benchmarks, just the only way to average *normalized* benchmarks. If you
have a good idea of your job mix, a weighted arithmetic mean of the raw data
is the way to go. Then you can normalize this mean to one machine if you feel
like it.

Regardless, the case *against* arithmetic means of normalized numbers 
seems completely incontrovertable, regardless of what one considers the
"best" replacement. It's not acceptable for the results to depend on 
which machine one arbitrarily decides to normalize to. 

The paper discussed here is:

  Flemming, Philip J. and Wallace, John J. "How not to lie with
    statistics: The correct way to summarize benchmark resuls",
    Communications of the ACM, Vol. 29, No. 3. (March 1986).


Radford Neal
The University of Calgary

In article <3600003@hplabsb.UUCP> wiemann@hplabsb.UUCP (Alan Wiemann) writes:
>                                                   Instead "[they] had the
>same person modify or write all the tests so [they] could be sure that the same
>algorithms would be used for all the processors" (page 56 of the IEEE article).
>Thus the benchmark results reflect not only the individual processors' ability
>to execute instructions but also the cleverness of this programmer in using
>each microprocessor's instruction set and architecture.  The results reported
>should not be considered true measures of the relative performance of these
>microprocessors.

The Computer Family Architecture (CFA) project of the Army and Navy in the
70's attempted to measure the performance of various computers. This is
considered, I believe, to be one of the ``classic'' benchmark efforts.

    "c	Programmers were not permitted to make _algorithmic_ improvements
	or modifications, but rather were required to translate the PDL
	descriptions into assembly language. Programmers were free to
	optimize their test programs to the extent possible with highly
	optimizing compilers. This ``hand translation'' of strictly defined
	algorithms was expected to reduce variations due to programmer skill."

	"Computer Structures: Principles and Examples" pg.58
	_Siewiorek,_Bell,_and_Newell_, McGraw-Hill 1982

If you are measuring the performance of a processor, then it is not
necessarily desireable to compile a standard program. If instead you
are measuring compiler/processor combined performance, then you
should certainly use the same source for all measurements. It is
much easier to rewrite a poor compiler than to redesign a poor architecture.

Kendall Auel
Tektronix, Inc.
(I don't claim or disclaim anything)

In article <3600003@hplabsb.UUCP> wiemann@hplabsb.UUCP (Alan Wiemann) writes:
>Benchmark comparisons give valid results only if the same program is presented
>to each machine.  The compiler is considered part of the "machine" and its
>performance contributes to the overall performance of the machine.
> [...]

I don't believe that this is a valid point.  The article benchmarks
*processors*, not systems.  While the compiler is part of a system, it is
not part of a processor.  You are correct that the skill of the assembler
programmer is quite important -- however, using a compiler would only have
raised an issue concerning the skill of the compiler writer.

What I found ludicrous in the article was that they found that (to paraphrase)
'an internal cache was quite helpful'.  The way they set up the hardware it
should have been pretty damned obvious that an internal cache would be helpful.
The I80386 results were not made making use of 2 cycle external memory accesses.
With the delays induced for external memory accesses, those machines with
internal caches were clearly superior.  I guess that helps a lot if you are
going to build a 16/20MHz M68020/I80386 system with ~150ns DRAM and no external
cache (ala Compaq).  But it sure doesn't mean much to most of the Unix boxes
out there.
-- 

Mark Campbell    Phone: (803)-791-6697     E-Mail: !ncsu!ncrcae!sauron!campbell

The quantity being measured can also affect the choice of the appropriate
mean!  E.g., are you measuring speed or time?  (An old puzzle is relevant:
After driving one mile at 30 mph, how fast do you have to drive the second
mile in order to average 60 mph?  The answer is *not* 90 mph - and the
arithmetic average is *not* correct in this context.  Hint: look at the
harmonic mean.)
I have seen benchmarks presented both ways - both in time (e.g. seconds) and
in speed (executions/second).
Additionally, several people have correctly pointed out that the benchmark
elements should be weighted to reflect the frequency of occurance of that
type of work in *your* job stream.  This is a very important point which
probably surpasses the importance of the harmonic/arithmetic/geometric mean
differences.

--henry schaffer  n c state univ

The geometric and harmonic means are MEANINGLESS for benchmarking machines.
They are only useful for fudging the results to try to make an
unbalanced, ill-designed, machine look good.

Look at it this way: if you've got two jobs to get done and
two machines take the following times:

	Machine A	Machine B
Job 1	10 sec		20 sec
Job 2	10 sec		5 sec

Does anyone disagree that Machine A takes 20 seconds,
and Machine B takes 25 seconds? It should be obvious
that machine A is the faster machine FOR THE GIVEN WORKLOAD.

Even if the times were more extreme:

	Machine A	Machine B
Job 1	10 sec		20 sec
Job 2	10 sec		1 sec

Still, Machine B is slower. It doesn't matter a damn
that Job 2 executes ten times faster on Machine B,
because it was so slow on Job 1 that it already lost
the race before even starting on Job 2.

Now, if you run Job 2 more often than Job 1, or Job 2
is more closely representative of the workload you
intend for the machine, then sure, go ahead and
adjust the weighting. However, the geometric
and harmonic means take neither of these factors into account,
and effectively use inconsistent weightings between machines.

Doesn't anyone remember the parable of the Tortise and the Hare?
I suppose someone will now try and convince us that
the Tortise should have been the winner via the geometric mean!

-- 

Craig Hansen			|	 "Evahthun' tastes
MIPS Computer Systems		|	 bettah when it
...decwrl!mips!hansen		|	 sits on a RISC"

In article <3600003@hplabsb.UUCP> wiemann@hplabsb.UUCP (Alan Wiemann) writes:
>Benchmark comparisons give valid results only if the same program is presented
>to each machine.  The compiler is considered part of the "machine" and its
>performance contributes to the overall performance of the machine.  This study
>did not present the same program to each machine.  Instead "[they] had the
>same person modify or write all the tests so [they] could be sure that the same
>algorithms would be used for all the processors" (page 56 of the IEEE article).
>Thus the benchmark results reflect not only the individual processors' ability
>to execute instructions but also the cleverness of this programmer in using
>each microprocessor's instruction set and architecture.  The results reported
>should not be considered true measures of the relative performance of these
>microprocessors.

How else do you compare assembly language performance between two machines with
different architectures?  Often critical sections of code will be coded in
assembler to increase speed.  The capability of the architecture to support
fast hand-coded assembler can have a significant effect on the performance of
the program.  So we need to do assembly language benchmarks.  I submit it is a
valid comparison to code the same algorithm into assembly on each machine.  
However, this coding must be done by and individual with equivalent experience
on each machine, spending the same amount of time programming.  That is, the
programmer is not allowed to bias the results by spending unfair amounts of
time optimizing his favorite processor.  The bottom line is that we must trust
the benchmarks, correlate them with other benchmarks, or do them ourselves.

By the same token, including the is often an invalid comparison because the
compiler can have a significant effect on the resulting performance.  Suppose
I take a student built, unoptimizing compiler for machine A and a highly tuned
optimizing compiler for machine B.  Now, if the two machines are anywhere close
in performance, machine B will win.  Here again, the bottom line is that we
must trust the benchmarks, correlate them with other benchmarks, or do them
ourselves.

Of coarse, we could have a competitive benchmark between interested parties.
Any takers?
-- 
  Lawrence Crowl		716-275-5766	University of Rochester
			crowl@rochester.arpa	Computer Science Department
 ...!{allegra,decvax,seismo}!rochester!crowl	Rochester, New York,  14627

In article <21344@rochester.ARPA> crowl@rochtest.UUCP (Lawrence Crowl) writes:
>By the same token, including the [compiler] is often an invalid comparison
>because the compiler can have a significant effect on the resulting
>performance.  Suppose I take a student built, unoptimizing compiler for
>machine A and a highly tuned optimizing compiler for machine B.  Now, if
>the two machines are anywhere close in performance, machine B will win.
>Here again, the bottom line is that we must trust the benchmarks, correlate
>them with other benchmarks, or do them ourselves.

The issue here is who is using the benchmarks, and for what?  If, as for most
of us, writing one's own compiler is not an option, then all that matters is
how our program will perform using the compilers available for it.  It
doesn't much matter if machine A is really faster than machine B, if the
only compilers available for machine A generate code which is so much worse
than that available on machine B that the programs actually run slower.

Frank Adams                           ihnp4!philabs!pwa-b!mmintl!franka
Multimate International    52 Oakland Ave North    E. Hartford, CT 06108