🚀 go-pugleaf

in <message id fiendishly hidden by brain-damaged software... sorry>
djbpitt@vms.cis.pitt.edu (Professor David J. Birnbaum) writes:

> I would also be interested in seeing some discussion of the issues
> raised in the posting to which I am responding.  But it is misleading
> to caricature the question of adequacy of coverage as quibbling over
> the presence or absence of individual characters.  The real issue is
> whether a multilingual character set should be able to support texts
> such as Professor Steensland's monograph.

> This issue is not a squabble about a single missing character with
> a funny name.  It is about how the issue of separable diacritics
> affects not only programming concerns; this issue affects in a sub-
> stantial way the adequacy of coverage of the character set.  And
> the latter must be paramount; we first decide what must be represented
> and only then can we evaluate the ramifications of one or another sys-
> tem of representation.  And what must be represented is written culture,
> not just vendors' databases of clients.

These arguments clearly represent several viewpoints which I must respectfully
disagree with:

A character set, like any data representation, is a combination of codes and
processing rules that together comprise an abstraction of properties of
real-world entities.  Every abstraction, no matter how well constructed, loses
some information.  E.g., no program on any machine will ever truly be able to
represent *all* integers.  To state that "adequacy of coverage... must be
paramount" is reflective of only one position in a spectrum of user
requirements for character sets.  Show me a machine that represents "all"
integers, and I'll show it an integer that it can't represent.  We
therefore compromise in our abstraction, and settle for a representation
that considers adequacy of coverage for the *vast majority* of the foreseen
users to be paramount.

I don't believe that, as worthy as such efforts are, works like Professor
Steensland's are the ones that govern the design of a character set.  Please
note that I did not say a character set *should not* provide coverage for
such a work.  I merely say that critical design decisions that effect the
methods by which *all* characters in a standard are represented should not
be made based solely (or paramountly -- I know there are linguists reading
this group, so if that isn't a word... mea culpa) on the "high-end"
requirements.  I believe that the processing needs of the 99.99+% of
users whose requirements fall short of this high-end of the spectrum
should play a larger role in the design of a new character set.

For some I'm sure this is sad but true: someone is going to pay for the
development and implementation of any new character set by any vendor.  So
please don't blame anyone for putting the written culture of the client
base ahead of the more challenging written culture of a few additional
"high-end" customers.  I speak from experience on this matter: my company
was a pioneer in bringing text processing to the mass market, with solid
customer bases in business, government, academics, etc. in the US and
abrod; yet for years my advocacy of larger character repertoires for
our customers was largely un-heard, because the work involved was too large
and the number of high-end customers willing to pay for the effort was too
small.  Colleagues with other vendors have told me that the same is true
in their situations.  (Please... don't insult me with "and look where your
company is today..."  Our character set coverage issues pale in comparison to
the things that have put us where we are now.)

I believe that the way to compromise on this question is to recognize two
very distinctly different classes of need: one is for *rendering* an extremely
wide variety of written symbols, and the other is for *processing* a less
extensive set.  Rendering Professor Steensland's work is a substantially
different problem from processing it.  I would argue that even if one
considers the community of users who need to *see* Professor Steensland's
work on line, the number whose efficient use of the information contained
within it would be constrained by an inability to search for (or otherwise
process) certain specific symbols is very small indeed.  Ergo, I suggest that
not everything that appears to be rendered as text needs to be processed
as text.

I emphatically am not suggesting that there be two character set standards
(one for rendering and one for processing).  I believe that the definition
of a character set should be influenced more heavily by processing needs
than by rendering needs, and that non-character (e.g. graphic or image)
representations of symbols required in the written culture of certain
linguistic, bibliographic, historical, or scientific works is a more useful
solution to real world problems.

For processing character data from a large repertoire, I believe that the
10646 design (flawed though it may be, too) is superior to Unicode's.  I
have many reasons, and I have informally passed them on to the Unicode folks.
But my objections go the very philosophy of Unicode, so I never expected any
reaction to them.  Unicode should be respected for what it is: a very thorough
synthesis of research and design efforts oriented towards solving the problem
of character rendering rather than that of character processing.  I just think
that's the wrong problem.

rich schwartz   (All views expressed are my own, and not Wang Labs, Inc.'s.)
 rschwartz@office.wang.com      VOICE (508) 967 5027     FAX (508) 967 0947
     Wang Labs, Inc., M/S 019-58A, 1 Industrial Ave., Lowell, MA 01851