🚀 go-pugleaf

>From daavid@marloo.cpsg.com.au Mon Jul 19 13:49:40 1993
Received: from kowari.cpsg.com.au by bohra.cpsg.com.au with SMTP
	(5.65c/16.2) id AA05658; Mon, 19 Jul 1993 13:49:37 +1000
Received: from marloo.cpsg.com.au by kowari.cpsg.com.au with SMTP
	(5.65c/DEC-Ultrix/4.3) id AA15693; Mon, 19 Jul 1993 13:49:31 +1000
Received: by marloo.cpsg.com.au
	(1.37.109.4/16.2) id AA01957; Mon, 19 Jul 93 13:48:37 +1000
Date: Mon, 19 Jul 93 13:48:37 +1000
From: Daavid Turnbull <daavid@marloo.cpsg.com.au>
Message-Id: <9307190348.AA01957@marloo.cpsg.com.au>
To: daavid@marloo.cpsg.com.au
Subject: UNICODE and NT
Status: O


>Path: kowari.cpsg.com.au!ariel.ucs.unimelb.EDU.AU!munnari.oz.au!news.Hawaii.Edu!ames!olivea!grapevine.lcs.mit.edu!ai-lab!wheat-che
>From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
>Newsgroups: comp.os.ms-windows.nt.misc
>Subject: Re: UNICODE and NT
>Date: 14 Jul 1993 02:18:58 GMT
>Organization: MIT Artificial Intelligence Lab
>Lines: 190
>Message-ID: <21vqeiINNlss@life.ai.mit.edu>
>References: <DVANDERR.93Jul7160213@hen3ry.oracle.com> <ROBERTK.93Jul9115406@rkrajewski.lotus.com> <CA25r9.Cs4@cbmvax.cbm.commodore
>NNTP-Posting-Host: wheat-chex.ai.mit.edu

[Darren Greenwald]
:
| When you start looking at these issues, it turns out the overriding
| issue is language identification, something which Unicode does
| not necessarily make easy.  Infact the unification of codes may
| make it very difficult to identify the language which the string was
| originally written in.

Unicode neither aids nor discourages language identification.  Language
identification *is not* a function provided by any standard character set
in use today.  If you are inferring that one can determine language
by virtue of knowing the character set being used, then you are mistaken.
I can encode English, Malay, Hawaiian, Swahili, and a very large number
of languages with ASCII, and I will not know which language I have by
looking at the characters alone.  And for that matter, I can encode these
also with JIS X 0208 too which will not give me a clue as to the
language being represented.  I cannot even with complete certainty assume
that I have Japanese when encountering a Kanji character encoded in JIS.
I can write a great deal of Chinese using JIS.

Of course you are correct that knowing the language of a text enables
certain types of processing; however, it is not the domain of a character
encoding to encode language.  If it were, then there might by as many
as 2,000 or more instances of the symbol <A> in a universal character set
which distinguished among languages.

| For example, looking at a filename written in Japan may be considered
| to be an English filename (if it has ASCII, which it probably
| will; some filenames may be entirely English while others may be
| English mixed with other languages), or a Chinese filename; the
| confusion could happen if the string happens to have used some
| of the unified kanji codes.

How can unification of Han characters influence the ability to
interpret a string as either English or Chinese?  Its pretty clear
which characters are English and which are Chinese.

| Likewise proper rendering of non-spacing diacritics requirees
| knowing which language writing system should be used; because
| of unification, you cannot arbitrarily seek into the middle of
| Unicode text (you could end up in the middle of a multi-code
| sequence, and may have to seek back to 'guess' at what writing
| system the code was originally written in to select glyphs properly
| for display.

This example doesn't seem to be coherent.  Unicode does not
specify the language of any character; that is, neither non-spacing
(combining) characters nor base (non-combining) characters specify
language.  Therefore, seeking forward or backward would not help
in any case.  If you are inferring the existence of a stateful
rich text mechanism (such as the embedding of language identifiers),
then you can't go seeking into the middle of a string to render it
any more than you can seek into the middle of an RTF file and expect
to render it.

| In fact, something as simple as converting the popular Japanese
| Shift-JIS to Unicode and back CAN be a problem!  Shift-JIS has
| this concept of single byte/double byte ASCII, SYMBOLS, and KATAKANA.
| Losing that information is very undesirable.

You are apparently not familiar with Shift JIS and Unicode.  Unicode
is a superset of Shift-JIS.  It is also a superset of the union of
Shift-JIS and ASCII.  To explain this simply, this means that there
exists a one-to-one correspondence between { SJIS, ASCII } & { UNICODE },
i.e., no information whatsoever is lost.  If you believe otherwise,
perhaps you would share the details of your belief (e.g., which characters
exactly do not map).  By the way, the whole issue of single byte vs.
double byte does not apply at all.  Perhaps you are confusing number
of bytes with visual width (e.g., zenkaku vs hankaku).  [Before anyone
asks, Unicode is a superset of the union { JIS X 0201 (8-bit), JIS X 0208,
JIS X 0212 }.]

| Unicode did not try to address these kinds of issues, but, to
| implement multi-language support at the OS level, the issues
| must be addressed.  That, or you make it an application specific
| problem.  The later choice just guarantees a great deal of
| incompatability, and need for private sequences which essentially
| means that Unicode buys nothing.

Au contraire!  One of the most important goals of Unicode was to
preserve the integrity of existing data.  Indeed, this goal operated
so as to override many of the ideals of Unicode (such as the taboo
on encoding presentation forms (glyphs)).

Actually the efforts of the Unicode consortium have been considerably
larger in the area of implementation than efforts made for previously
developed character encoding standards.  How many other character set
development efforts have sponsored implementor workshops on an
international scale?

You are correct that OS vendors have a signficant responsibility in
developing Unicode capable systems.  But you seem to imply that it
might have been otherwise.  Do you really think that a character set
can, by its very design, solve all the problems of i18n and multilingual
processing?  A very sound principle in software development is
limiting the scope of your work.  This principle is also known as
divide and conquer.  The division of labor chosen by the designers of
Unicode was explicit and based on many years of experience in the
development of solutions for multiple language processing.  Unicode
is a character set; its users (operating system developers, application
developers, etc.) must provide the real solutions for its use just
like they provided for ASCII and all other character sets.

| Don't misunderstand.  I have a great deal of respect for the
| Unicode committee for all the work they have done, but I wonder
| if they were misguided in two regards:
|
| 1.) They seem to have been motivated to create a smaller
| version of the ISO DIS standard.
|
| 2.) They created a codeset in a vacuum; I am not sure separating
| the problems of implementation from the codeset was wise, except
| that it allowed them to get work done.

Assuming you are referring to the original DIS 10646, dated 4 November
1990, it is useful to point out that Unicode (1.1) contains all the
characters in that DIS plus many more. Indeed, Unicode covers a much
larger collection of scripts than did the original DIS.  One reason
that Unicode became the basis for the BMP of the final ISO/IEC 10646-1:
1993 is that it was deemed to be superior in terms of its coverage of
different scripts (and thus, of different written languages).

As for creating "a codeset in a vacuum", I must say that it appears
that you have very little knowledge of the development (or the
developers) of Unicode.  I would hardly call the designers of the
Xerox Character Set, the Association for Font Information Interchange,
the designers of the Macintosh Kanjitalk and Mac Arabic systems,
the character coding experts of Digital, IBM, Hewlett Packard, or
the developers of bibliographic character set standards inexperienced
in the matters of coding or developing implementations for multiple
language platforms.  Would you?

| It turns out that there was an alternative solution which would
| have been smaller than the ISO standard, and yet provided all that
| Unicode gives us.  That is a solution which includes language
| identifiers (or actually codeset identifiers), and use of
| an agreed upon codeset for each identified writing system.

There was (and is) no such alternative.  If someone has dreamed one
up, you might as well tell them to forget it.  There is slightly less
than a 0% chance that it will go anywhere.

| If I pass in a string containing Arabic, but do not have an Arabic
| font, just what does Windows going to do with it?  What do I have
| to do to get the text displayed, and just what is the predominate
| display direction for the users system?  What about text entry?
| How do I obtain a specific glyph (say for the purpose of rotation,
| or stand-alone display) for languages which have positional
| forms like Arabic?

As it currently stands, neither Windows NT or Windows supports the
full range of requirements needed to actually use Unicode in its
entirety.  However, that doesn't mean that it won't support it in
the future.  I fully expect Microsoft to develop more inclusive
support for the various languages which are adequately represented
with Unicode (a number significantly larger than anyone has ever
seen on one system before).

The key to the whole puzzle is having a stable, adequate, and *single*
character set which provides for the needs of representing the basic
content of multiple languages the world over.  And the name of the key
is Unicode (or for standards-minded people, ISO/IEC 10646-1:1993 UCS2).
By supporting the *mere* representation of Unicode data, Windows NT
goes far beyond what existed previously.  Now we must gently (or
with a big stick, as you will) encourage Microsoft to develop all the
support needed to actually use it in its full glory.  [Of course
the same applies to all other Unicode-based system suppliers.  I'm not
deprecating subset implementations of Unicode; however, blanket claims
of "supporting Unicode" can vary widely in their meaning, and anyone
making such blanket claims should probably be questioned closely.  The
Unicode consortium is now developing a procedure whereby Unicode based
systems (or applications) suppliers can make more clear to their users
(and themselves) exactly what portion of Unicode they actually support
and the degree to which they support it.]

Lest you mistake my response as overly critical, I'd like to say that
much of your posting is valuable and correct.  However, your perception
of Unicode seems to be based on incorrect information.  Perhaps you (and
others) would like to particpate more active in the Unicode Consortium?
It is open to membership with three classes of membership:  corporate,
associate, and individual.  Unlike many consortiums, Unicode strongly
encourages the participation of anyone with expertise in this area.
Indeed, it is the individual experts that seem to be the most active
participants much of the tiime.  Information about the consortium
and membership in it can be obtained from Steven Greenfield at
<unicode-inc@hq.m4.metaphor.com>.

Glenn Adams
Technical Director, Unicode Consortium

--
Daavid Turnbull AARnet: daavid@cpsg.com.au   Phone: +61 3 243 2300
CP Software Group  (ACN 006 640 133)         Fax:   +61 3 243 2301
	4th Floor, 493 St Kilda Road, Melbourne, Victoria, Australia, 3004