🚀 go-pugleaf

mccall@mksol.dseg.ti.com (fred j mccall 575-3539) writes:

>In <bhayden.725514513@teal> bhayden@teal.csn.org (Bruce Hayden) writes:

>>charlie@umnstat.stat.umn.edu (Charles Geyer) writes:

>>>In article <1992Dec22.034142.14471@fsl.noaa.gov> bear@kestrel.fsl.noaa.gov
>>>(Bear Giles) asks:

>>That is what is known at regression testing. First, many don't do it,
>>and second, even those that do often miss things. However, I would
>>contend that regression testing is a minimum requirement when upgrading
>>a product. .

>All true, but it still won't find all the bugs.  It won't find any of
>the ones that got through your initial tests and it won't necessarily
>find new ones that are introduced by the fixes you've put in.  What
>you seem to be arguing for is exhaustive testing (an impossibility) to
>be repeated each time a change is made.

No - I am not suggesting that everything be tested whenever a fix is
added. I am also not advocating finding all of the bugs...
But - I have spent enough of my life coping with bugs introduced 
when code was fixed to realize that more testing is required than most
companies provide.

I am suggesting two things really. First when a product is released,
or a new version thereof, prior to release a QC level should be d
determined, and stuck to. This means so many bugs of a certain level.
Secondly, a test suite should be developed as you go along, testing
in particular for previous bugs. Then the test suite is run against the
new version, making sure that old bugs don't resurface.
.
>You want it WHEN ???

Well, this is a combined marketing/management/legal problem. Invariably
marketing, for any number of reasons promises what cannot be delivered.
This will continue to happen until companies discover that the cost of
prematurely releasing software outweighs any marketing advantages gained.
Currently that is not the case. But it may be changing. 

In litigation over the software, one of the first things that competent
counsel will look at are the procedures actually followed in releasing
the software. Was due care exercised in designing, developing and
maintaining the software? Was QC in place, and was it followed?
(Or was it preempted by marketing?)

>>>A customer can't really complain if software doesn't work on hardware the
>>>vendor doesn't claim it works on.

But that's rarely the problem. More likely is the situation where the
software will run on a "generic" PC. Or maybe the vendor didn't specify
what the software would run on (or wouldn't). 

I will be the first to agree that a vendor should not have liability for
situations where they have prominently disclaimed (before purchase) 
running on a specified configuration, or have stated that they only run
on certain configurations. But that rarely happens. 

>>Well, if the vendor says that it will only work on a limited configuration
>>you may be right. However, I think that it is suicide to write software
>>(at least for a PC) that runs on a limited range of machines. I ask what
>>is wrong with a company if they can't get their product to run on a 
>>generic PC (or if they won't even try). 

>Define 'generic PC'.  Most people define it as "what is on my desk".
>Unfortunately for software producers, that's not the same for
>everyone.  

The question from a legal point of view is - whose problem is it?
I would argue (see below) that it's the problem of the vendor, and
not the person depending on the vendor's expertise. 
,
>>My experience is that few software vendors spend much time defining
>>what their software will not run on (except maybe to say it requires
>>at least DOS 3.10, or Windows 3.1). My feeling is that if they don't
>>tell me before I buy a piece of software (i.e. on the box) that it 
>>won't run on my machine, I expect it to run.

>They don't spend a lot of time doing it for a number of reasons.  1)
>We're back to that problem of exhaustive test.  There are just too may
>possible combinations of hardware and software to try them all.  2)
>DOS typically doesn't stress the hardware all that much, and will let
>hardware with sloppy timing happily run.  Software that is more
>efficient and makes better use of the hardware will require tighter
>timings on it, and will fail on some machines.  Windows 3.1 fails on
>some machines with marginal memory that Windows 3.0 runs on, for
>example.  OS/2 fails on even more, in part because it tries to really
>use the whole machine instead of only using the bottom 640k of memory
>and then getting to the rest of it via kludges.  3) Software can
>interact with other software.  What TSRs you have loaded AND THE ORDER
>THEY ARE LOADED IN can make a big difference, but this will vary with
>what kind of hardware you have, what memory manager you use, etc.

Well, that doesn't answer the question. If I have what I "reasonably"
consider a generic PC (and a not too wild configuration), and I buy
a piece of software that allegedly runs on a PC, then I (as the buyer)
can reasonably expect that software to run on my machine. If it doesn't,
then the liability must rest on the party most able to prevent the 
problem - the vendor, and not on the buyer, who is depending
on the expertise of the vendor.

That means that vendors should (and probably will) be held to a standard
of either disclaiming support for a specified configuration or machine,
or supporting it. The disclaimer should also probably be visible to
any purchaser before he buys the software (and not burried within)
.
>Just how many version would you have them support, and for how long?
>On how many different machines in how many different configurations?

As for how many versions - the question is not relevant. My position
is that if you are going to upgrade - you should make sure that you
do it right - or face the consequences (in particular legal) for your 
failure.

As for different machines and different configurations - the questions
is also relatively easy. If you have had a problem with a particular
machine and/or configuration, then you should either 1) disclaim
prominently responsibility for any future levels and/or versions, or
2) test with that problem machine and/or configuration. 

>>Again, if the vendor did legitimate regression testing before releasing
>>the upgrade, you wouldn't have this problem. 

>Well, you wouldn't have it AS MUCH.  However, from reading your
>comments, it just doesn't seem that you've considered what kind of
>real-world problems you're talking about.

Well, I was in software development long enough (15 years) that I 
have seen how it is done wrong more times than I can count. I have
also seen it done right. My experience is that all it takes to do
it right is proper planning and adequate resources. 

I think that we will see in the very near future that courts start
imposing liability for defective software, if the reason that the
software is defective is that proper software engineering methodology
was not followed. 

This means that an expert witness will come along and say that if
(for example) this procedure, etc. (such as QC) had been done, the
program would probably not have failed as it did. Then (s)he would
go on and testify that that procedure (or whatever) is taught in
many (most) software engineering classes. Stating that the procedure
was ignored or overridden because of marketing decisions has rarely
been a credable defense. 

Bruce E. Hayden
(303) 758-8400
bhayden@csn.org