🚀 go-pugleaf

Glenn Adams writes:
> The most pernicious of the problems, #1 above, can now be solve fairly well
> thanks to the new ISO/IEC 10646-1:1993 Universal Character Set standard (and
> Unicode, the more fully specified 2-byte form of this new standard).  However,
> problems remain in the actual use of Unicode; e.g., the possibility of having
> a NUL octet embedded in a 2-byte character encoding.  [In Unicode, a coded
> character must be interpreted as a full 16-bit wide character, and not as
> a string of octets such as are found in multi-byte encodings like JIS.]

Rob Pike and Ken Thompson have proposed a very elegant solution
to the "Unicode problem" in their recent "Hello World or..."
paper. They present a synchronizable byte-stream encoding that
uses one bit of each byte to indicate whether it carries an
ascii character (0) or is part of a three-byte sequence that
encodes a Unicode character (1). The next one or two bits in
the bytes of each 3-byte sequence encode position, leaving
6+5+5=16 bits for the Unicode information. This guarantees
that each byte's place in the stream can be identified from
its content, and ascii - including the ascii NUL - stays ascii.

				Adam_V_Reed@ATT.com