🚀 go-pugleaf

  In article <1992Sep8.113126.1@cbrown.claremont.edu> dhosek@cbrown.claremont.edu writes:
  >In article <27618@life.ai.mit.edu>, glenn@wheat-chex.ai.mit.edu (Glenn A. Adams) writes:
  >> 32-bits are *not* needed, 16-bits suffices for representing the primary
  >> content of *all* languages, both modern and archaic.  Of course, 32-bits
  >> might be useful for glyph encodings; but Unicode doesn't encode glyphs.
  > 
  >> [Note: 16-bits only suffices when a compositional approach is allowed,
  >> such as is the case with Unicode and ISO10646; i.e., to represent the
  >> textual information displayed as LATIN CAPITAL G WITH CIRCUMFLEX, the
  >> two Unicode characters LATIN CAPITAL LETTER G and NON-SPACING CIRCUMFLEX
  >> are employed.  There is no need to encode all combinations of formal
  >> elements as single encoding units.]
  >
  >The problem is that text-sorting algorithims are greatly
  >complicated by having multiple characters representing a single
  >sorting unit. This is a general problem of course (for example,
  >in Spanish, ch, ll and rr are each treated as single letters so,
  >e.g., lo comes before llegar in an alphabetized list of Spanish
  >words, on the other hand, in an American English dictionary, i
  >and i-with-diaresis would be treated as the same letter for
  >sorting purposes). The approach of coding diacriticals separately
  >has both advantages and disadvantages and I don't think that
  >it's quite the closed book represented here.

Spanish is only one of *many* languages that sort digraphs, trigraphs,
or even n-graphs as atomic units.  Since it is a general problem, one
has to solve it anyway.  And, I would suggesst that one *cannot* treat <i>
and <i-with-diaresis> as the same letter; for if you did, you could not
reliably sort text, i.e., different ordering of the same input may
produce different output orders:  the first requirement for a sorting
algorithm is predictability.

It is not even as easy as you suggest, for sorting French correctly
requires not only separating diacritics, but reversing their order in
a secondary ordering pass.  Then in Vietnamese, some diacritics are
considered as part of a letter for sorting (circumflex, breve, horn),
whereas others are separate (acute, grave, hook-above, tilde, dot-under),
the latter representing the tones SA<'C, HUYE>!N, HO?I, NGA", and NA<.NG,
respectively.  In Arabic, Hebrew, and Indic scripts, diacritical marks
represent vowels, tones, cantillation (chanting) signs, silent letters,
and other kinds of orthographic signifiers, all of which must be taken
into account in sorting, and which must be treated separately from the
base letter to which they are applied.  Correct linguistic sorting of
text cannot, and should not, be overloaded onto the semantics of character
encoding.  The purpose of encoding is not to make any one text process
most efficient, but to make it such that the most important text processes
can *adequately* perform their tasks.  For example, it would be easier to
do hyphenation of English text if each syllable were encoded separately;
it would be easier to do case insensitive equality testing if CASE were
encoded separately.  It is easier to display Arabic if glyphs are encoded,
though it makes searching very difficult.  And so on...

When I say that 16-bits are adequate, I mean that this suffices for the
most important text processes, namely, display, searching, sorting, and
parsing.  The full generality of the writing systems of the world requires
that more sophisticated models be developed for each of these processes
than currently holds with ASCII-encoded English text.  But none of these
*requires* 32 bits of encoding space in order to perform their task over
the range of the world's writing systems, past, present, and future:
16-bits will suffice.