🚀 go-pugleaf

leonard@qiclab.scn.rain.com (Leonard Erickson) writes:

>bhayden@teal.csn.org (Bruce Hayden) writes:

>>eoliver@ralph.cs.haverford.edu (Erik Oliver) writes:

>>I think that depends on how common the hardware and hardware combination
>>is. If the hardware is anywhere close to common, then I would contend
>>that it is a reasonable to test a major product on all of the common
>>hardware. 

>The problem is that the *combinations* are unmanageable. Say the 10 most
>common motherboards, the 10 most common video boards, the 10 most common
>TSRs. Plus the 10 most common add-on devices. That gives you *ten thousand*
>combinations, assuming only one selection from each class. But some can
>exist in multiples. And you actually need to test for more than the
>10 most common in some classes. 

That is not exactly what I am proposing. Sure 10*10*10 = 10,000. 
But - practically, your program is probably either going to work,
or not work, with a given TSR. Testing each with the 10*10 motherboard/
video board combinations is absurd.

Now lets look at the motherboard/video board problem. First, there
are a number of motherboard/video-board combinations that are locked
in. For example, IBM tends to use IBM motherboards and IBM video boards.
Is it realistic to test IBM motherboards with Compaq video boards?.
Secondly, is it realistic to test each type of motherboard, or more likely
each type of BIOS? There are many fewer manufactures of BIOS than 
of motherboards. Which is it that causes the problems most often with
programs? I think you will find it is the BIOS. 
r
>It *isn't* practical to test even *most* interactions. All you can reasonably
>do is test the some "standard" configurations. (Ones that are known to 
>be "the standard" that everything else is supposed to work like)

Like what? What is the "Standard" anymore? Surely not IBM with MCA.
(Do you have MCA? I sure don't, and don't intend to. I don't
want to pay double for all my addons.)

>>I also would contend that once you have serious problems with a specified
>>hardware combination (with previous releases or products) then you may
>>potentially have a duty to duplicate that hardware combination for future
>>QC testing.

>Or just list that setup as one that the software *won't* work with.

Again, I think that you are being simplistic. Much better to determine
first what the problem really was. Was it really a software problem, 
or was it a problem with that hardware? Also look at exactly where
the problem arose. Did it arise between the program and the BIOS?
Or between the program and the video board. 

If the problem is an anomoly with a very rare video board, then sure
disclaim support. But, if the problem is with something more common
such as the BIOS, then its better to fix your software, and thereafter
test with that piece of hardware or firmware.

There are two reasons to fix the software and test for that combination
thereafter. First, cutting yourself off from owners of a certian BIOS
for instance will cut you out of a significant part of your market.

Secondly, from a legal point of view, any such disclosure must be 
prominent and visible on the box of the software. If you wait until
someone has opened the box and is looking at the shrink-wrap license,
or indeed looking at anything on the inside of the box, it is too late.

>>Obviously, you have a resource problem here. IBM can afford to basically
>>test almost anything imaginable (so why didn't OS/2 work on many non-IBM
>>platforms?) A small company can probably only afford to test on a limited
>>number of platforms and combinations.

>>I disagree. We have the capability today to set an acceptable bug level
>>before the release, then not release until we have reached that level.
>>That isn't happening in many cases. Either we aren't setting the level,
>>or if we are, we are shipping before we get to that point.

>I know of one company that is trying to do this, and you should hear
>the users scream about the fact that the new version isn't ready. 

And why are the users clammoring for the new version? Did the company
maybe engage in a little preannouncement marketing, getting everyone's
interest up so they wouldn't abandon the product for a competitor?

There are two things that go into new versions and releases - new
features, and fixes. Stability fixes should be regression tested
as a matter of course. That leaves features and enhancements. This
is where marketing gets a company in trouble by preanouncing products
long before they are ready. Often long before the company really
knows how long they will actually take to code and test.

Bruce E. Hayden
(303) 758-8400
bhayden@csn.org