🚀 go-pugleaf

In article <4dgq46$so5@news.csus.edu>, will_yu@sfsu.edu (WILLIAM C YU) writes:
|> Chad Irby (cirby@gate.net) wrote:
|> 
|> : The whole idea of the Spec benchmarks was supposed to be a non-fudgable 
|> : result, right?
|> 
|> Which fantasy world did you come from?
|> 
|> : Put it simply.  Intel cheated on Spec92.  What makes you think they were 
|> : honest in running the code for Spec95?
|> 
|> Everybody cheats at benchmarking.

|> As I recall, the rule of the Spec benchmark is (roughly) "anything goes". 
|> Think about it carefully -- if a benchmark had restrictions on what 
|> types of instructions and optimizations are allowable, how could anyone 
|> produce a fair set of guidelines?
|> 
|> : You know, for years people have been saying that Intel machines turned
|> : out real-world results much slower than their supposed "benchmark"
|> : numbers would indicate.  There's your answer.
|> 
|> Earth to Chad. The Spec benchmarks have never been an accurate 
|> representation of real world work for any architecture. What the Spec 
|> numbers show is the maximum theoretical performance of a machine.

Reality to YU:
1) You have a made a bunch of firm statements.

2) Some of those statments are factually incorrect ... which then casts
doubt on some of the opinions you have also expressed.
[Note: I'm one of the original founders of SPEC & have done benchmarking
studies for years before that.]

3) Once upon a time, SPEC benchmarks, used as they were supposed to be
[i.e., report all the numbers, look at the variance, match specific ones
to actual codes], were certainly useful measures, and they correlated
pretty well with what we saw on other real codes.  It will take a while
to figure out how well SPEC95s correlate.

4) Benchmarks always get "gamed" over time; some vendors do more of this
than others; not everyone cheats [and I personally know a lot of the people
involved; do you?] SPEC has plenty of rules: it is *not* "anything* goes,
contrary to your assertion. Some of us have even vetoed some kinds of
optimizations even when we had them and no one else did, because we
believed they were misleading in this context.


-- 
-john mashey    DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP:    mash@sgi.com 
DDD:    415-933-3090	FAX: 415-967-8496
USPS:   Silicon Graphics 6L-005, 2011 N. Shoreline Blvd, Mountain View, CA 94039-7311