🚀 go-pugleaf

I'm requesting this message to be posted to this net.
I am not a member. -JR

My original comments that I've heard about being published, once 
again, contains some inaccuracies.  My intent at that time, some 
18 months ago, was to stir up some controversy and present the 
'con' view point regarding the results being published in popular 
magazines and on local BBSs.  That may or may not have been a good 
idea, since I'm still trying to explain myself.  At any rate, I 
also recently read S. Brecher's comments dated Nov. 8 on yet 
'another network' rebutting the old feud once again.  Steve's 
corrections to my statements are accurate.  The old discussion 
that Steve and I had from times of old gave me more to think 
about.  His rebuttal corrected some of my inaccuracies.  I've 
since learned of the errors in that original article.  This 
original article taken out of context cannot be appreciated unless 
the entire discussion was also presented.  I wish to withdraw the 
original article due to mistakes it contained, and after my re-
examining of the Disk Timer source code.  Although, this has not 
changed my opinion of Disk Timer.

I do have more than 'no idea' of what I'm talking about, but my 
misunderstandings republished recently may lead one to believe 
otherwise.  Then about 12 months ago, another fellow decided to 
revise Disk Timer and bring forth a new version.  I had a long 
series of discussions on yet 'another network'.  The following is 
my comments made at that time, which I still believe true.  [I 
made a few new edits, shown in brackets]  I still stand my 
following statements made during the second discussion of the 
continuing Disk Timer "Pro vs. Con" series.

________________________________________
Date:     Mon Sep 8, 1987  10:45 am  EST 

I wish to express my sentiments regarding the resent proposal of 
revising DiskTimer.  Before I make my comments, I'll add that I 
have great respect for Steve Brecher's work.  My first reaction 
was, "Why do we need another DiskTimer?"  The earlier versions 
caused more misunderstandings and misrepresentations than anything 
else.  Benchmarks are not to be considered lightly.  As an 
example, consider how BYTE magazine can't produce a valid set of 
tests for the IBM vs. Macintosh benchmarks.  There are issues that 
must be addressed if we are going to be publishing the results of 
any benchmark test.  It must consider the methods of how a system 
operates in real life.  So, if DiskTimer is to be considered at 
all then it needs to test a drive under "real world" situations.  
This means it is going to require standard file I/O (e.g. using 
the Mac's File Manager).  Without the use of the test files, the 
results are too far removed from reality.

In other words, the purpose of a drive is to read and write files.  
Any benchmark of a drive's performance must test its ability to do 
this, which is done via the File Manager.  Steve has defended the 
non-use of the File Manager by stating, "I developed DiskTimer II 
precisely to avoid using File Manager calls, i.e., to provide a 
benchmark that could be run without initializing the disk".  This 
is ridiculous statement in my opinion.  How many people can use a 
hard disk that hasn't been initialized? [text deleted -JR]

I understand that DiskTimer is actually testing the driver's 
lowest level of the interface, so then maybe it should be renamed 
to "DriverTimer".  I also understand that DiskTimer is not 
intended to be a test of actual system performance.  But then, of 
what use are the results in that case?  The fact is that DiskTimer 
is going to be scooped up by everyone with a hard disk and the 
results are going to be publicized at every user level.  (Witness 
the 'special' version of DiskTimer that SuperMac Technologies has 
produced.  They even claim that their drives are three times 
faster than everyone else's based on these results.)  Some 
magazine editors have stated that they will no longer publish 
performance tests based on DiskTimer results.  (refer to Ric 
Ford's recent article of drives for the Mac SE printed in MacWeek)  
MacWeek has found that the timings of DiskTimer II have no 
correlation with real world performance, and therefore the results 
have little or no value in their comparisons.

The following list is requirements that the new DiskTimer (DT) 
must include, if it is to be rewritten at all.

1.   There needs to be a differentiation in the test between 
multiple vs. single block transfers.  A large multiple block reads 
will always perform best on a drive formatted with a 1:1 
interleave.  In the real world of Macintosh, most drives are 
interleaved.  This causes the multiple block read/write tests to 
be weighted towards any drive formatted with no interleave and 
will handicap a drive that is interleaved. [as we move to higher 
capacity drives, the 1:1 interleave is more common since these 
drives often do not support interleaving -JR]

2.   In the real world of Macintosh, all file I/O requests are 
given by the File Manager.  If DT is to ignore reading and writing 
files stored on the disk, then its results are invalid [in my 
opinion -JR].  Testing the performance of a drive requires the 
testing of the drive under typical conditions.  Since the biggest 
percentage of the over-head in file I/O is caused by the File 
Manager, then DT III must use it.  DT should create, read, and 
write files.  Yes, DT results will then be subjected to a drive's 
fragmentation.  But then again, the FileManager will try to use a 
contiguous space on the drive, which is another reason to use the 
FileManager.  This is the typical situation.

3.   Consideration must be given to the drive's interleave, number 
of read/write heads, number of bytes per block, number of blocks 
per track, and number of tracks per cylinders.  Without this 
information, no test will be fair to drives with dissimilar 
geometries.

4.   DT needs to address drive configurations that can invalidate 
its results, such as a drive with integral caching.  I feel there 
is an inconsistency in the philosophy of DT.  Steve says, "the 
results in such cases will almost certainly be so low (so 'good') 
as to immediately identify them as invalid."  But DT is being 
distributed to the public at large, which does not have the 
technical knowledge to determine when the results are too 'good' 
to be true.  The typical user of DT is Joe Blow trying to beat the 
guy next door.  [this is the reason I take an issue with Disk 
Timer -JR]

5.   I recommend a larger than 24K file to be used in the test.  
The majority of files on a Macintosh system are considerably 
larger than this.

6.   During the test, a warning should be presented to the user 
telling him that interrupting it may cause loss of data, and 
possibly the entire drive.  Although this is true with any and all 
programs that write to a drive, DT is unique.  It quietly sits 
there performing its test.  If a less than knowledgeable user gets 
bored with the test, he may be tempted to abort it by resetting or 
powering off the computer.

7a.   I've saved the criticism of the access time test for last, 
because it has the biggest problems.  With DT's access test, a 
slower seeking drive may test better than a faster seeking drive.  
How?  The answer is in the drive's geometry.  As an example; if a 
drive with 15 read/write heads is matched against a drive with 
only 4 heads, then the bigger drive has access to nearly four 
times as much data without performing any head seeks!  This is 
exactly why DT's access testing methods are not a valid test of 
head seeks.  The drive with only 4 heads will need to travel a lot 
more distance seeking in DT's test.  Any access test needs to 
insure that an actual head seek across a specific number of tracks 
has incurred.  If DT III is going to report access times, then it 
better make certain that the heads have actually moved a certain 
number of tracks.  I've been able to alter the results of the 
access test by changing the interleave of the drive.  Obviously, 
the drive's access time could not have actually changed by this.  
Also, consider the SCSI Seek command in the access test.

7b.   Steve defends his access time results by stating, "Do not 
confuse DiskTimer II's access time test with an attempt to 
simulate hardware vendors' average access time statistic."  Then I 
beg, please, do not call it an "access time" test.  The current 
access time test reads a block and then another block one Megabyte 
into the drive, therefore adding to this result the time it takes 
to read the two blocks it is seeking to.  Steve believes "this 
confoundment (sic) is not significant".  But in fact it could very 
well be, considering the interleave of the drive.  The heads may 
have found the proper track, but the block was not in position 
under the heads.  This will add the time it takes to spin the 
platter(s) to get the block under the heads.  So the problem 
described here is testing seeks across so many Megabytes.  Without 
knowledge of the drive's geometry, this testing method is not 
useful and potentially misleading.

In conclusion, the reason I'm so adamant to the proposed revision 
to DT is that it has been exploited by far too many people.  It 
has established its name as 'the' test to use.  It is the de-facto 
standard, but it is a loaded weapon and in the wrong hands it is 
dangerous.  Mis-information is worse than no information.  I feel 
if DT is to be re-written then it needs to be re-philosophized.  
As Ephraim Vishniac has pointed out, DT has been exploited by 
certain vendors.  Certainly everyone will agree, the purpose of a 
drive is to store files.  With this in mind, any performance 
results of a drive should test the drive's ability to transfer 
files to and from a Macintosh.  DiskTimer in the past has only 
tested the driver's interface.  This is too far removed from the 
real world of hard disk usage.


My final words regarding this 'benchmark war' is this.  People
interested in the evaluation of SCSI drives on the Macintosh need
a copy of SCSI Evaluator.  Send $20 to "Digital Microware, PO Box
3527, Mission Viejo, CA 92690.  In return you will receive a 60+
page manual discussing benchmarks for SCSI drives.  Before 
continuing any further discussion of SCSI test programs, everyone 
concerned needs to read the information in the SCSI Evaluator 
manual.  That is all.  -JR