🚀 go-pugleaf

In article <4j96jp$m7r@news.erie.net>, pce@moose.erie.net (Paul
Christenson) wrote:

> When are people going to learn that you can't directly compare benchmarks 
> across platforms?  Especially when the different systems use different 
> processor families?
> 
> For instance, Linux comes up with a "BogoMIPS" rating on bootup.  It's an
> indication of how fast Linux did some timing loops.  One friend's 486DX4-100
> has a BogoMIPS rating of 50.  Another friend's Pentium-133 also has a 
> BogoMIPS rating of 50.  Does that mean that the two machines are the same
> speed?  Of course not.
> 
> A friend of mine compiled up the BogoMIPS-determining code on a high end
> SGI machine.  The machine is overall about 10x faster than the DX4-100, 
> yet its rating was LOWER than that of the DX4.
> 
> Bottom line...  Benchmarks are useless between different platforms.

OK. Let's quit comparing computers. For that matter, the next time I need
to buy a car, I'll just buy the first one I happen to stumble
across--after all, you can't compare different cars. Gee, we don't need to
spend millions of dollars on elections either. How do you compare
candidates. We'll just assign a random number generator the task of
allocating votes.

You HAVE to compare platforms. How else are you going to make a decision.

All you've shown is that BogoMIPS is a lousy benchmark. If you use good
benchmarks, you get reasonably consistent performance stories. In general,
the best benchmark is to run the application the computer user will need
to use most often. The other benchmarks can be evaluated on how they
approximate real life performance.

Most of the standard benchmarks are at least self-consistent. That is, a
P-6 is faster than a P5 which is faster than a 486 and so on. They also
consistently show that a 604 is in the same league as a P6 and a 601 is in
the same league as a P5.  You could argue that finer distinction is
impossible, but to argue that benchmarks have no use is a silly argument,
IMO.

-- 
Regards,       Joe Ragosta

Copyright Joseph M. Ragosta, 1996. Non-exclusive, royalty free
license to distribute this post granted to any service provider 
except Microsoft. By posting this, Microsoft agrees to pay $1,000 per 
posting.

In article <jragosta-2703960819210001@ppp-2016.dca.net> jragosta@dca.net (Joe Ragosta) writes:
>In article <4j96jp$m7r@news.erie.net>, pce@moose.erie.net (Paul
>Christenson) wrote:
>
>> When are people going to learn that you can't directly compare benchmarks 
>> across platforms?  Especially when the different systems use different 
>> processor families?
>> 
>> For instance, Linux comes up with a "BogoMIPS" rating on bootup.  It's an
>> indication of how fast Linux did some timing loops.  One friend's 486DX4-100
>> has a BogoMIPS rating of 50.  Another friend's Pentium-133 also has a 
>> BogoMIPS rating of 50.  Does that mean that the two machines are the same
>> speed?  Of course not.
>> 
>> A friend of mine compiled up the BogoMIPS-determining code on a high end
>> SGI machine.  The machine is overall about 10x faster than the DX4-100, 
>> yet its rating was LOWER than that of the DX4.
>> 
>> Bottom line...  Benchmarks are useless between different platforms.
>
GEE, maybe you are right!  I WAS thinking about putting MY benchmark on the 
net.  It breaks things down into 9 catagories, rather than one neat number.

loops
32bit int
double
term I/O
reads from disk cache
random writes
random reads
sequential writes
sequential reads

Each catagory, on my P5 has a maximum error rate of +-4%.  Because it does each
for a given time, it will run on VERY fast systems, or the slowest system I have
ever heard of.

OF COURSE, the disk cache COULD be cache memory, and the disk COULD be very
fragmented or clean, and the compiler has a lot to do with other measurements,
etc.....  But if one computer does 3 million loops, and another does 2 million,
the first is almost SURELY faster than the second, and it CERTAINLY is when 
using the same compiler.

SO, benchmarks aren't ENTIRELY useless.  SURE, they're not perfect, but they
are better than subjective measurements.

Steve

Joe Ragosta wrote:
> 
> In article <4j96jp$m7r@news.erie.net>, pce@moose.erie.net (Paul
> Christenson) wrote:
> 
> > When are people going to learn that you can't directly compare benchmarks
> > across platforms?  Especially when the different systems use different
> > processor families?
> >
> > For instance, Linux comes up with a "BogoMIPS" rating on bootup.  It's an
> > indication of how fast Linux did some timing loops.  One friend's 486DX4-100
> > has a BogoMIPS rating of 50.  Another friend's Pentium-133 also has a
> > BogoMIPS rating of 50.  Does that mean that the two machines are the same
> > speed?  Of course not.
> >
> > A friend of mine compiled up the BogoMIPS-determining code on a high end
> > SGI machine.  The machine is overall about 10x faster than the DX4-100,
> > yet its rating was LOWER than that of the DX4.
> >
> > Bottom line...  Benchmarks are useless between different platforms.
> 
> OK. Let's quit comparing computers. For that matter, the next time I need
> to buy a car, I'll just buy the first one I happen to stumble
> across--after all, you can't compare different cars. Gee, we don't need to
> spend millions of dollars on elections either. How do you compare
> candidates. We'll just assign a random number generator the task of
> allocating votes.

You completely missed the point.  He's not saying one should not compare
computers, he's saying that a benchmark is not going to give you an accurate
comparison.  If you are, you won't be comparing apples with apples (I hate
to use stupid cliches, but I couldn't think of a better one).

> You HAVE to compare platforms. How else are you going to make a decision.

Exactly.  Don't look at benchmarks.

> All you've shown is that BogoMIPS is a lousy benchmark. If you use good
> benchmarks, you get reasonably consistent performance stories. In general,
> the best benchmark is to run the application the computer user will need
> to use most often. The other benchmarks can be evaluated on how they
> approximate real life performance.

Okay, a bit of background, here.  BogoMIPS is *not* a benchmark (Bogo stands
for bogus).  Like he said, it's a timing loop.  Linus made it up so the kernel
would have a realatively consistent timing loop on any computer.  That, my
friend, is a very lousy benchmark, indeed, because it's not one.

> Most of the standard benchmarks are at least self-consistent. That is, a
> P-6 is faster than a P5 which is faster than a 486 and so on. They also
> consistently show that a 604 is in the same league as a P6 and a 601 is in
> the same league as a P5.  You could argue that finer distinction is
> impossible, but to argue that benchmarks have no use is a silly argument,
> IMO.

Of course they'll be consistent!  All that says is that all 601's are built
the same.  A good benchmark on a P6 may not account for differences in
the architecture between it and a 601.  Or how a fast processor won't get
you anywhere in a slow computer.  And, by the way, there really is no way
to compare a P6 and a 604.  It depends on what you're looking for.  It's
like deciding whether to buy a BMW or a Cadillac (assuming you can actually
afford either, or would even want to); you will inevitably compare the numbers,
but you know it won't matter.  Even if the BMW has more acceleration (higher
benchmark numbers, to aid those missing the parallel between the two), you
know the cadillac has a bigger engine, and could probably haul more weight,
if called to do so.  What really matters is whether you want American Luxury
or German engineering.

Just like what really matters with the computer is whether you want the proven
GUI of a Mac, or the flexibility of the PC (even if the GUI is somewhat less
impressive, and definitely not as stable).