🚀 go-pugleaf

  Date: Fri, 11 Jun 93 16:37:59 +0200
  From: Roy Badami <roy@harlequin.co.uk>

  I haven't been paying much attnetion to 10646/Unicode for a while, so
  I'd like to catch up with what's been happening recently.  When I last
  looked at this, unicode was 16-bit (with unified han characters, and
  trailling diacritics), and 10646 was 32-bit bit with encoding rules
  for generating a sequence of octets (which were upward compatible with
  straight ascii text)...

ISO/IEC 10646-1:1993, recently published by ISO CS/ITTF, and the upcoming
Unicode 1.1 define the same character repertoire and assign identical names
and codepoints to the 34,168 characters in this repertoire.  The remaining
characters of the Base Multilingual Plane of 10646 (also known as UCS-2, for
2-octet universal coded-character set form) are 6,467 reserved positions
and 24,901 positions left for future assignment; 6,400 of the reserved
positions are available for private use.  All major modern scripts are
currently covered except Burmese, Ethiopian, Khmer, Mongolian, Sinhalese,
and Tibetan.  These will be added in the near future as soon as their
encoding details are finalized.  Many other historical and minority scripts
are also being evaluated for encoding in a future version.

Unicode is indeed a 16-bit encoding (rather than a two byte encoding).  A
NUL byte (0x00) may appear in either byte of the 16-bit encoding.  So it
is not directly compatible with software operating on null-byte terminated
"char *" strings.  Microsoft NT uses wchar_t * as the type for a null
character (not byte) terminated Unicode string, defining wchar_t as an
unsigned short.

10646 defines to encoding forms, a 16-bit form, known as UCS-2, and a 32-bit
form, known as UCS-4 (also called the canonical form).  UCS-2 and UCS-4
encodings must be interpreted as unsigned short and unsigned long types
respectively.  They cannot be interpreted as char * byte strings.  The
MSB 16-bits of the UCS-4 is divided into equal 8-bit parts, designating
the GROUP (bits 31-24) and PLANE (bits 23-16), respectively.  The MSB of
the GROUP octet must be zero, so only 128 groups are available.  When
the GROUP and PLANE octet of a UCS-4 form are zero, that designates the
Base Multilingual Plane (BMP).  UCS-2 is an abbreviated form of a UCS-4
BMP encoding.  Unicode 1.1 and UCS-2 are code for code identical.  The
only diffference is in the conformance clause.  Unicode defines additional
semantics to characters that aren't defined in 10646, such as directionality
for every character.  In addition, 10646 defines 3 implementation levels
which serve to limit the characters supported by an implementation.  Level
0 contains no combining characters.  Unicode 1.1 does not have implementation
levels, so it is effectively level 3, although it allows arbitrary subsetting
which can have the same effect.

The intent of combining characters is that their image graphically combines
with another character to produce a (composite) graphical symbol.  However,
the result of this combination is not a "character" as defined by 10646.
Therefore, both 10646 and Unicode are closed "chararcter" repertoires.
However, they are open repertoires in terms of graphical symbols, since no
limitations are placed on graphical symbol composition (other than by the
implementation levels in 10646). Neither 10646 or Unicode specify what
"support" or "implementation" means with respect to combining characters,
although they informally state that "the intention of combining characters
is that they graphically combine with a non-combining character or a
sequence of a non-combining character and one or more combining characters."

A simple level 3 implementation may choose to treat combining characters as
if they were non-combining, i.e., as normal spacing characters.  This may
produce undesirable visual display or processing in some cases; however,
this is preferable (in my opinion) to excluding level 2 and level 3 data.
Nothing in Unicode or 10646 prevent such an implementation from being
conformant, since the manner of processing combining characters is not
specified in a conformance clause.

The problems with interchanging and processing 16-bit and 32-bit encoding
forms may be alleviated through the use of a transformation format.  Two
formats have been defined (one in 10646), UTF-1, and another by JTC1/SC22/WG15,
UTF-2.  The latter is currently used by the AT&T Plan 9 operating system as
its native encoding.  The UTF-2 encoding transforms a sequence of UCS-4 (or
UCS-2) encodings into an octet stream in which each ASCII character is
represented by itself (including C0 control characters), and where all other
characters consist of a variable number of octets with bit 7 set.  Thus
all of UCS-2 characters fit into 1, 2, or 3 octets.  The encoding of UTF-2
is such that randomly branching into such a string allows reliable
synchronization (either forward or backward) with a character encoding
boundary (unlike, e.g., Shift JIS DBCS systems).

Source code for routines which implement both UTF-1 and UTF-2 may be found
by anonymous FTP on METIS.COM [140.186.33.40].  See README file in /pub.

  I'm sure that a lot must have happening since then -- could someone
  point me at some sources of info, or provide me with a brief summary
  of the current state of standardization...?

You can obtain further information about Unicode from Steven Greenfield,
<unicode-inc@hq.m4.metaphor.com>.  You can also subscribe to the list
<Unicode@Unicode.org> (send subscription requests to
<Unicode-request@Unicode.org>.  Other information can be obtained by
anonymous FTP from UNICODE.ORG.

ISO/IEC 10646-1:1993 should now be available from your national standards
body.  If they don't have it yet, they should contact ISO Central
Secretariat in Geneva.

Glenn Adams
Technical Director, Unicode Consortium