🚀 go-pugleaf

In article <4664@sun.uucp>, rmarti@sun.uucp (Bob Marti) writes:
> 
> There is a set of benchmarks for DBMS developped at U of Wisconsin by a guy
> named DeWitt.  The DeWitt benchmarks seem to be a de facto standard for
> relational DBMSs.  I seem to recall that a year or two ago, INGRES performed
> substantially faster than Oracle according to the DeWitt benchmark, although
> Oracle claimed that the benchmarks were unfair, since DeWitt -- then visiting
> professor at UC Berkeley -- supposedly used a souped up version of INGRES
> which was not available on the market at the time.

The true story is much worse than that.  The numbers quoted for Oracle in
the DeWitt paper were for a pre-release of Oracle Version 3.1 on a Berkeley
4.1c release of UNIX.  Since Oracle does not normally run in on Berkeley
UNIX (owing to the lack of shared memory) a large electronics manufacturer
coerced us into give them the sources so they could port Oracle to that
environment ostensibly for the purpose of running some of their own benchmarks.
Next thing we knew customers were calling us up quoting the DeWitt paper.

Because the paper compared Oracle (pre-release 3.1) on 4.1c (an operating
system which could not properly support it) with a well tuned University
Ingres on 4.1 with Commercial Ingres (2.0) on VMS [All on different hardware]
we consider it to be a good example of how NOT to do a benchmark.

This is not to say that the benchmark itself is not a reasonable (though
arbitrary) test of RDBMS performance.  One problem with it is that it
doesn't measure concurrancy performance or the impact of the RDBMS on other
system applications.

The last test we ran comparing Oracle on VMS to Ingres on VMS we were
beating Ingres in something like 13 out of 16 tests in the DeWitt Benchmark.
I would hesitate to post numbers to the net since I'm not sure how the
other RDBMS manufacturers would respond.

Our general sense here is that Oracle performs much better than Ingres on
micro's (3B2), somewhat better on mini's (3B20,VAX) and somewhat poorer on
mainframes (Amdahl/UTS).  We tend to outperform Informix and Unify across the
entire range for medium to complex queries.  The performance of Ingres and
Oracle are tending to approach their limits as increased efforts lead to
diminishing returns (or good returns in very narrow areas).

I have a proposed Benchmark Specification (based on DeWitt) from a company
in Palo Alto, CA (International Data Corporation Technology Laboratories)
which attempts to address the concurrancy and system impact issues.
In theory they intend to run it on all of the RDBMS.  The only problem
is that it requires the dedication of several machines for several
weeks in order to run it and that is alot of time (money) to run tests which
are likely to be obsolete in 6 months given the evolution of these systems.

There is also the problem that all of the RDBMS have different interface
languages so you have to write a seperate benchmark for each system meaning
you are to a degree comparing apples with oranges.  Until everyone conforms
to the ANSI X3H2 SQL standard it will be very difficult to realistically compare
one RDBMS with another.

-- 
Robert Bradbury
Oracle Corporation
(206) 364-1442                            {ihnp4!muuxl,hplabs}!oracle!bradbury