🚀 go-pugleaf

Submitted-by: karish@pangea.Stanford.EDU (Chuck Karish)

In article <1halvbINN9kd@ftp.UU.NET> stephe@usenix.org
(Stephen R. Walli) writes, as part of a well-considered
analysis that points out some critical problems:

>I think POSIX is caving in under its own weight.

I agree.

>Test Method Madness

>Now, a standard cannot offically exit balloting without having a test
>method specification that is also a standard.  This instantly sets up
>a directly competing body of text to the original standard.

I've been under the impression that the test methods must
always defer to the semantics defined in the base document.
The competition for developers' attention is real!

>Test methods standards will become the annointed specification for the
>test suite to demonstrate conformance by organisations with the funds
>or market presence to demand as much. Implementations can hit the
>narrower mark of the test suite (embodying the standard test methods)
>to naively certify rather than hit the standard itself.

This is always the case, as far as I can tell.  Implementations
of a standard are tested (and used!) according to their behavior.
It is problematic whether aspects of the base specification
that are not testable belonged there in the first place.

>If you don't
>realise the subtle and nasty differences that can appear, spend some
>time with the POSIX.1 standard (IEEE Std 1003.1- 1990), and with its
>newly declared standard test methods (IEEE Std 1003.3.1-1992).

It's to be called 2003.1-1992.  Is the problem with the
many errors in 2003.1 or with the philosophy of
certification based only on testable behavior?

My observation has been that the nasty differences arise
because the writers of the base standard don't consider
carefully enough how the standard will be implemented.  The
discipline of considering test methods helps focus
attention on these problems before errors in the standards
are immortalized as hacks in the implementations.

>Now what happens when someone requests an interpretation of a standard
>with its test methods?

In the past, people who wondered about the meaning of an
aspect of a base standard asked the relevant interpretations
committee for a ruling.  I don't see that the situation
is much different now.

Each assertion in a test methods document is a very specific
statement about the meaning of the base standard.  The
interpretations committee for the base standard should be
able to assess assertion accuracy easily.

>If the request is leveled against the base, what guarantees are there
>that the test methods, i.e. a separate standard, will be kept
>synchronized?

People who use test suites will either demand it or will
volunteer and do it themselves.  From a practical
viewpoint, it's a tremendous amount of work for a
certifying authority to continually make allowances for
errors in test methods; it's much easier to fix them.

This requires that an active ongoing interpretation and
maintenance process be supported.

>Do test methods need to be standards?

If the test methods aren't standardized, test suites will
become de facto standards without adequate review.  I don't
see a viable alternative to standardizing the test methods.
I hope we can continue to move away from the practice of
hacking our implementations to work around inadequacies in
test suites, just as every implementation of a specification
no longer has to be a bug-for-bug clone of a reference
implementation now that we have a process to standardize
specifications.

>Testing is
>expensive, but the market ultimately protects itself. What has been
>done in the TCP/IP space? (If you don't think TCP/IP is a successful
>widely implemented specification, stop reading now.)

This example is not completely analogous to the POSIX operating
system standards.  Communication systems are worthless without 
complete interoperability, so the marketplace is quite efficient
in requiring conformance.

Move away from that a bit and look at specifications like SCSI,
NFS, and the One True GUI, and you'll see that the marketplace
does not always cause implementors to see the need to compromise and
converge.

>What about the C
>language?  No one specified a set of test methods for the ANSI C
>standard. People in the know wanted to see how to test the C standard,
>and through a lot of hard work built the Plum-Hall test suite. The
>U.S. government created a FIPS for C, and chose an available suite.
>There were no test methods for this work. No added burden on the
>volunteer standards community to respecify itself.

Here's where politics become a problem.  It's difficult to
design test methods and implement a test suite that do not
favor one implementation over another.  Moreover, it's
expensive: sufficiently expensive that it's too risky to
develop a test suite that may not be accepted.  This is
problematic even with standardized test methods.

[ The solution: don't ballot on test methods. ]
>Ballot groups could concentrate on the real specification in front of
>them. Repeat again: Bad test methods standards will be dangerous in
>the marketplace.

Without the balloting, there won't be any test methods
standards.  And if we think back to the misgivings expressed
above about naively certifying against test suites rather than
against standards, we'll come to expect the bad test methods to
drive out the good.


The question of who will be willing to take on all the work
of doing the interpretations is a good one.  The 1003.1
interpretations folks will be swamped for some time to come
with questions about the accuracy of 2003.1 assertions.
The situation would be an absolute nightmare if they were
in the position of having to answer corresponding sets of
questions relating to multiple test methods specifications
that had not been through a review process.

It's much more efficient to get a standard right the first
time than by patching it through the review and maintenance
process.  Without balloting on test methods, we'll wind up
with more wasted effort and less to show for it, both in
the base standards and in the test methods.

--
Chuck Karish          karish@mindcraft.com
(415) 323-9000 x117   karish@pangea.stanford.edu

Volume-Number: Volume 29, Number 90