🚀 go-pugleaf

In article <4d7mln$f940@hopi.gate.net>, Chad Irby <cirby@gate.net> wrote:
>Robert Salasidis (salasidis@medcor.mcgill.ca) wrote:
>: 
>: The Intel benchmark error was only for the SpecInt92 #s, the SpecInt95 
>: and fp were accurate.
>
>Were they?
>
>Gee- after Intel managed to make their "reference compiler" run 10% 
>faster with a "bug," it kinda makes you wonder just how much of their 
>benchmark-generating code is designed to do nothing else but run 
>benchmarks. 
>
>The whole idea of the Spec benchmarks was supposed to be a non-fudgable 
>result, right?
>
>Put it simply.  Intel cheated on Spec92.  What makes you think they were 
>honest in running the code for Spec95?
>
>You know, for years people have been saying that Intel machines turned 
>out real-world results much slower than their supposed "benchmark" 
>numbers would indicate.  There's your answer.
>
>-- 
>
>Chad Irby           /    My greatest fear: that future generations will,
>cirby@gate.net     /      for some reason, refer to me as an "optimist."

Was it not Andy Grove himself, CEO of Intel who said, "Business is war",
and, "Only the paranoid survive." ?

Caveat emptor.

David Derbes [loki@midway.uchicago.edu]

Chad Irby (cirby@gate.net) wrote:

: The whole idea of the Spec benchmarks was supposed to be a non-fudgable 
: result, right?

Which fantasy world did you come from?

: Put it simply.  Intel cheated on Spec92.  What makes you think they were 
: honest in running the code for Spec95?

Everybody cheats at benchmarking.

As I recall, the rule of the Spec benchmark is (roughly) "anything goes". 
Think about it carefully -- if a benchmark had restrictions on what 
types of instructions and optimizations are allowable, how could anyone 
produce a fair set of guidelines?

: You know, for years people have been saying that Intel machines turned
: out real-world results much slower than their supposed "benchmark"
: numbers would indicate.  There's your answer.

Earth to Chad. The Spec benchmarks have never been an accurate 
representation of real world work for any architecture. What the Spec 
numbers show is the maximum theoretical performance of a machine.

In article <4dgq46$so5@news.csus.edu>, will_yu@sfsu.edu (WILLIAM C YU) writes:
|> Chad Irby (cirby@gate.net) wrote:
|> 
|> : The whole idea of the Spec benchmarks was supposed to be a non-fudgable 
|> : result, right?
|> 
|> Which fantasy world did you come from?
|> 
|> : Put it simply.  Intel cheated on Spec92.  What makes you think they were 
|> : honest in running the code for Spec95?
|> 
|> Everybody cheats at benchmarking.

|> As I recall, the rule of the Spec benchmark is (roughly) "anything goes". 
|> Think about it carefully -- if a benchmark had restrictions on what 
|> types of instructions and optimizations are allowable, how could anyone 
|> produce a fair set of guidelines?
|> 
|> : You know, for years people have been saying that Intel machines turned
|> : out real-world results much slower than their supposed "benchmark"
|> : numbers would indicate.  There's your answer.
|> 
|> Earth to Chad. The Spec benchmarks have never been an accurate 
|> representation of real world work for any architecture. What the Spec 
|> numbers show is the maximum theoretical performance of a machine.

Reality to YU:
1) You have a made a bunch of firm statements.

2) Some of those statments are factually incorrect ... which then casts
doubt on some of the opinions you have also expressed.
[Note: I'm one of the original founders of SPEC & have done benchmarking
studies for years before that.]

3) Once upon a time, SPEC benchmarks, used as they were supposed to be
[i.e., report all the numbers, look at the variance, match specific ones
to actual codes], were certainly useful measures, and they correlated
pretty well with what we saw on other real codes.  It will take a while
to figure out how well SPEC95s correlate.

4) Benchmarks always get "gamed" over time; some vendors do more of this
than others; not everyone cheats [and I personally know a lot of the people
involved; do you?] SPEC has plenty of rules: it is *not* "anything* goes,
contrary to your assertion. Some of us have even vetoed some kinds of
optimizations even when we had them and no one else did, because we
believed they were misleading in this context.


-- 
-john mashey    DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP:    mash@sgi.com 
DDD:    415-933-3090	FAX: 415-967-8496
USPS:   Silicon Graphics 6L-005, 2011 N. Shoreline Blvd, Mountain View, CA 94039-7311

In article <4dgq46$so5@news.csus.edu>, will_yu@sfsu.edu (WILLIAM C YU) wrote:

[]
> Earth to Chad. The Spec benchmarks have never been an accurate 
> representation of real world work for any architecture. What the Spec 
> numbers show is the maximum theoretical performance of a machine.

Uh, I think what they really show is how hard the vendor is willing to
work in order to look good on SPEC. That performance may, or may not, have
any relationship to performance on other code.

Nice how everyone is still avoiding the key question: Why does P6 look
twice as fast as a 604 on SPEC (tests run by Intel on Intel reference
hardware with an Intel reference compiler), while on Bytemarks, which are
tests that can be run independently by anyone, the 604 is almost twice as
fast at P6!!

Here is one idea: Why don't the good folks at Intel run the Byte tests
with their nice "reference" hardware and "reference" compiler and post the
results on  the Intel web site? Then we would know several things: (1) Is
the reference hardware/software good for anything other than SPEC? (2) How
fast/fancy is the reference hardware compared to, say, the HP P6 machines
will Watcom (or pick your own favorite) C compiler. This would be a useful
thing to do. As best I can tell, no one but Intel can do this, because no
one else has the reference hardware and compiler!