🚀 go-pugleaf

Gregory L Hansen wrote:
> 
> In article <4jfs0h$mer@daily-planet.nodak.edu>,
> Nathanael J Henderson <nahender@prairie.NoDak.edu> wrote:
> >Gregory L Hansen (ghansen@isc.sjsu.edu) wrote:
> >
> >: Instead of arguing about which platform is faster,  let's prove it with
> >: our own benchmarks, the UseMarks.  We'll publish standardized programs
> >[...]
> >
> >   You *couldn't* have posted this with a strait face!  :-)  To think
> >that ONE loop using only a handful of operations is anything aproaching a
> >credible benchmark!  :-)
> 
> Maybe I should have mentioned this before, but I want to make sure
> everyone understands I am anything but a professional programmer.  I'm a
> rank amateur.  I wouldn't know a good benchmarck if it spat out of my
> printer.
> 
> But after all this arguing about other people's benchmarks, it seemed
> like a great idea if we had our own.  But I thought it would be kind of
> lame if I just suggested it and didn't do anything to start things
> going.  Every single person is very welcome to criticize, introduce their
> own tests, even change the basic format and scoring, in productive ways
> that I would not have suspected.  In fact, I'm counting on it.  I just
> want to make sure everyone using them understand exactly where they came
> from and can reproduce them on any machine they can get their hands on.
> I'm hoping our tests will be done on more hardware than the others have
> been, too.

#pragma getting on soap box

Lest people forget, the SOLE purpose of a benchmark is to measure something
that has meaning.

This does not mean that if you call a program a benchmark that it defacto has
meaning.

The problem of performance metrics is in general very hard.  You can make some
statements about specific problems and how various things effect the results, but
beyond that there is not much else you can say.

In general, the most general benchmarks which reduce the system tested to a single
number are often the worst benchmarks, as they gloss over (or completely obscure)
the features and advantages of one architecture/chipset over another.  They hide
details.  Benchmarks are SUPPOSED to be all about exposing the details.

Then the meaning question comes in.  How relevant is the process of adding 1 million
random numbers together versus solving the problem for which you purchase the
computer?  That is, if you are going to take the time to look at the "benchmarks",
why not do a real set of benchmarks, measuring the time to complete operations of
relevance,  the ease of use of the system (often completely overlooked by those
doing the benchmarks, to the chagrin of those receiving the machines), the visual
quality and accuracy, etc.

Simply measuring speeds and feeds is not meaningful unless you operate in that
particular paradigm...  However, most people (who do not operate in that manner)
tend to accept the speeds and feeds picture on faith as representing a useful data
point...  arguable at best.

#pragma getting off soap box

-- 
+-----------------------+----------------------------+     /
|     Joe Landman       | +1 810 615 2169  voice     |    /
|   Systems Engineer    | +1 810 478 3181  fax       |   /
| Silicon Graphics Inc. |        x 5-7670  corporate |  /
|   24155 Drake Road,   | landman@detroit.sgi.com    | /
|  Farmington MI 48335  |   Mail stop DMR 225        |/
+-----------------------+----------------------------+