🚀 go-pugleaf

Please post!  I'd like to know too.
-- 
Pierre LEWIS    +1 514 765-8207      ESN 852-8207
Internet: lew@bnr.ca                 Corwan: lewis@bnrmtl

Implementing standards is like walking on water -- both work best when frozen.

The following answers are based off of second-hand info, and possibly
not correct due to faulty memory, so take with a grain of salt :-)


mskuhn@immd4.informatik.uni-erlangen.de (Markus Kuhn) writes:

>- How is ISO 10646 structured? Is it a 16 or a 32 bit character set?

I believe it's 32, since 65,536 characters aren't enough to support
EVERY language (especially the Chinese and Japanese character sets)
unless you leave out a bunch.

>- What subsets are defined in ISO 10646?

Not sure what you mean by subsets.  Language sets?  If so, it'll
(or, _should_) support all the characters available (European
(not hard, since mostly the same), Asian, Arabic, and any other
one I forgot :-)  No, more like every character set for every
country that is likely to have computer systems.  Does that make
sense? Or am I babbling...

>- How does it relate with US-ASCII, ISO Latin 1 and Unicode?
>- What's the standard's status today (DIS?)? When will it be published
>  officially?
>- What requirements are there for application programs (editors, etc.)
>  and libraries if they what to handle ISO 10646 files?
>- Has anyone already written a short ISO 10646 FAQ list with answers?
>- What else should a computer science student know about ISO 10646 
>  in the future?
No idea.  However, it will present some difficulties for programmers
since they (especially non-Asian ones) tend to think of strings
as being made up of 8 bit entities.  The Japanese and Chinese already
use 16 bits, so they might have somewhat an easier time converting

I also heard there might be prefix codes to determine the language
to be selected.  I'm not sure how that'll be implemented though--does
that mean there will be lots of holes in the 32 bit character map?

On the other hand, that'll make converting from native formats easier
(i.e. American set could be prefix xxx, with standard ASCII codes
following two 0's  ->   'A' = prefix 00 00 41 (hex))  The Japanese
I am sure, would like that a lot :-)  Save them a lot of work

-Mark

>Markus

>-- 
>Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
>Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
>------------------ ISO?  Nicht immer. Aber immer öfter! -------------------
-- 
---
Mark "Emerson" Chadwick -- ncc1701@acm.rpi.edu    chadwickmo@space.laafb.af.mil

In article <q++ygqb@rpi.edu> ncc1701@acm.rpi.edu writes:
>The following answers are based off of second-hand info, and possibly
>not correct due to faulty memory, so take with a grain of salt :-)
>
>
>mskuhn@immd4.informatik.uni-erlangen.de (Markus Kuhn) writes:
>
>>- How is ISO 10646 structured? Is it a 16 or a 32 bit character set?
[ much deleted ]
It's been a long time since I looked at 10646, but it was using a
scheme for 8, 16, and 32-bit encodings, and a code-page-select kind of
thing that gives you most of the 32-bit space for *each* language
(think of more like an encoding for a font rather than a character
set).

Also, because there was so much emphasis on the emerging Unicode
scheme, the ISO decided that 10646 and Unicode should be brought into
a single specification, so that all of Unicode would be a proper
subset of 10646.  (This entails some shuffling of small details on
both sides, of course, and will take some time to iron out.)  I
believe that the revised standard will come up for vote again around
the end of the year.

If you're *really* interested, the place to ask is your local National
Standards Body (ANSI, DIN, AFNOR, BSI, etc.).  They will (usually) be
happy to provide you with a current draft, if you claim that you have
a "material interest" in the standard.  (Your local NSB may have
slightly different rules, check with them.)  Usually, there is a
copying and handling fee, say US$ 50-150 range, depending.  I think
that 10646 really comes in several parts; one is the textual
explanation of how the encoding works, the others are the "glyph
sheets" that show what each character is supposed to look like for
each code in a particular code-page.  (You can imagine that this could
go into thousands of pages in total; you don't want *all* of them, do
you?  After all, this is supposed to cover almost every character ever
devised by man since writing began!)

And yes, since many C programs don't even handle 8-bit chars quite
correctly  (only 7 bits), there will be a *lot* of software that will
need to be rewritten extensively.  (And things like toupper() and
tolower() don't mean much for Kanji and many other writing systems.)
















-- 
Fritz Whittington     Texas Instruments, 12840 Hillcrest #200, Dallas, TX 75230
fritz@csc.ti.com                Office: +1 214 917 7437    FAX: +1 214 917 7487
_Since I am not an official TI spokesperson, these opinions contain no spokes._

Mark O. Chadwick <ncc1701@madoka.its.rpi.edu> writes:
|
|   The following answers are based off of second-hand info, and possibly
|   not correct due to faulty memory, so take with a grain of salt :-)

Oh, please, we have enough of nth-hand information about things already.

|   mskuhn@immd4.informatik.uni-erlangen.de (Markus Kuhn) writes:
|
|   >- How is ISO 10646 structured? Is it a 16 or a 32 bit character set?

Both.  You select which with an escape code or a "signature".  I think
16 bits is going to win the market hands down.  Chinese, Japanese and
Korean was "unified", and requires about 36000 bit combinations (or code
points in IBM-speak).

There's also a transformation code defined which makes it possible to
use only the bit combinations dedicated for graphic characters in other
ISO standards, corresponding to the 8-bit integers 32 through 126 and
160 through 255.  This is strictly a transformation code, though, and
internal representation should be the real 16-bit character numbers.

|   >- What subsets are defined in ISO 10646?

A number of subsets are defined, and can be requested with an escape
code.  I have only the DIS list, and more may be defined.

|   >- How does it relate with US-ASCII, ISO Latin 1 and Unicode?

US ASCII is found at positions 00xy (hex), where 0 <= x <= 7, and 0 <= y
<= F.  ISO Latin 1 is found at positions 00xy, where A <= x <= F, and y
as above.  Unicode and ISO 10646 have merged, and Unicode 1.x (x >= 1,
don't remember which) will be the same as ISO 10646.

|   >- What's the standard's status today (DIS?)? When will it be published
|   >  officially?

It passed the DIS to IS vote.  I don't know when it will be published
officially.

|   >- What requirements are there for application programs (editors, etc.)
|   >  and libraries if they what to handle ISO 10646 files?

I'm not qualified to reply to that.  (In other words, I'm still trying
to figure it out, and don't want to confuse anybody.)

|   >- Has anyone already written a short ISO 10646 FAQ list with answers?

I'm tempted, and will probably end up doing it, but don't hold your
breath.

|   >- What else should a computer science student know about ISO 10646
|   >  in the future?

Acquire it, or wait for my book on character sets and character data
representation, but you should definitely not hold your breath for that
one to come out!

|   No idea.  However, it will present some difficulties for programmers
|   since they (especially non-Asian ones) tend to think of strings
|   as being made up of 8 bit entities.  The Japanese and Chinese already
|   use 16 bits, so they might have somewhat an easier time converting

Programmers should generally be able to switch from "think 8 bits" to
"think 16 bits", but there are lots of nits and grits to get right, and
I'm still researching this topic.  The use of combining characters,
where you use a sequence of characters to make up one "character", will
pose _interesting_ problems.  The Chinese and Japanese have actually
been using two septets (for a total of fourteen bits), and this is a
true 16-bit coded character set, so they will face problems both with
the encoding, and the bit combinations corresponding to an individual
ideographic character.

|   I also heard there might be prefix codes to determine the language
|   to be selected.  I'm not sure how that'll be implemented though--does
|   that mean there will be lots of holes in the 32 bit character map?

The language chosen affects such things as collating sequences and such.
You can't use the subset mechanism for this purpose, and language is
really not the province of character sets in the first place.  There are
a lot of holes in the 16-bit character set.  The coded character set is
defined for 0000xxxx (0 <= x <= F), only.

Language would probably have to be provided as part of the system
environment by other means.  Standard C and POSIX provide the "locale"
mechanism, which may or may not be sufficient for this purpose.

The above represents the best of my knowledge and comes from reading the
ISO working papers on the standard development, voting on it, reading
the ISO10646 mailing list and its archives, and I have no reason to
think any of it is wrong or that I misrepresent ISO 10646.  For the one,
true answer, though, consult the standard.  I'm sure comp.std.internat
will be the choice newsgroup to announce its availability.

Best regards,
</Erik>
--
Erik Naggum             |  ISO  8879 SGML     |      +47 295 0313
                        |  ISO 10744 HyTime   |
<erik@naggum.no>        |  ISO 10646 UCS      |      Memento, terrigena.
<enag@ifi.uio.no>       |  ISO  9899 C        |      Memento, vita brevis.

Fritz Whittington <fritz@ra.csc.ti.com> writes:
|
|   I believe that the revised standard will come up for vote again
|   around the end of the year.

This is incorrect.  Voting terminated on 1992-04-10, and ISO DIS 10646
became ISO 10646, a fully official International Standard.  It has not
been published and made available, yet that I know of.  Partly due to
the size of the standard, partly due to the number of comments with
votes, which I think have been reconciled by now (with the exception of
Turkey's irrational request to replace ISO 8859-1 in the 00xx "row" with
their favorite set, and Japan's more reasonable concerns).

|   (You can imagine that this could go into thousands of pages in
|   total; you don't want *all* of them, do you?  After all, this is
|   supposed to cover almost every character ever devised by man since
|   writing began!)

... and then some.  The DIS was actually "only" 710 pages.

Best regards,
</Erik>
--
Erik Naggum             |  ISO  8879 SGML     |      +47 295 0313
                        |  ISO 10744 HyTime   |
<erik@naggum.no>        |  ISO 10646 UCS      |      Memento, terrigena.
<enag@ifi.uio.no>       |  ISO  9899 C        |      Memento, vita brevis.

In article <BstGEq.7E7@immd4.informatik.uni-erlangen.de>,
mskuhn@immd4.informatik.uni-erlangen.de (Markus Kuhn) wrote:

> - How is ISO 10646 structured? Is it a 16 or a 32 bit character set?

32, although there is a 16-bit form that includes only those characters in
the so-called Basic Multilingual Plane.  Currently, 10646 contains no
characters outside of the BMP. 

> - What subsets are defined in ISO 10646?

I don't believe there are any subsets formally defined right now, although
there is provision for "the specification of subsets of coded graphic
characters for use in interchange, by originating devices and by receiving
devices."  

> - How does it relate with US-ASCII, ISO Latin 1 and Unicode?

10646 code points 0000 0000 through 0000 007F are identical to 7-bit ASCII.
10646 code points 0000 0000 through 0000 00FF are identical to Latin 1.
10646 code points 0000 0000 through 0000 FFFF (the BMP) are identical to
Unicode 1.1.  (Version 1.1 of Unicode will be issued later this year.)

The main differences between 10646 and Unicode are:

10646 is a 32-bit character set which can exist in a 16-bit form.  Unicode
is a 16-bit character set.  

Although the BMP of 10646 and Unicode are code-point identical right now,
Unicode specifies additional semantics and behavior for characters
(particularly visual behavior) which are not in 10646.

> - What's the standard's status today (DIS?)? When will it be published
>   officially?

10646 passed its final vote last month and WG2 met in Seoul to dispose of
various countries' comments.  At this point, the editor has to determine
exactly what changes need to be made to bring it into accordance with the
resolutions made during the Seoul meeting and some administrative details
have to be taken care of, but the voting process has ended.  The IS will
probably be published early next year.  It will be printed by AFII and so
should look really nice.  

> - What requirements are there for application programs (editors, etc.)
>   and libraries if they what to handle ISO 10646 files?

The single biggest bugaboo is the fact that 10646 characters include NUL
bytes in them.  Don't assume that a NUL byte terminates a string, and don't
treat 10646 text as a series of bytes.  C programmers will have to get used
to using types other than char to represent text.  

This particular problem will, perhaps, hit UNIX-based systems harder than
others but is not insuperable.  AT&T is working on one solution which uses
the transformation format specified in the DIS that makes it possible to
represent and manipulate 10646 as a series of bytes.  

> - Has anyone already written a short ISO 10646 FAQ list with answers?

I've seen one or two short summaries but I'm not sure right now where they
might be hiding.  

If you want detailed information on 10646 but don't want to wait until the
IS is issued, you may want to look into acquiring the Unicode Standard, v.
1.0.  It's available in two volumes from Addison-Wesley (ISBN's
0-201-56788-1 and 0-201-60845-6; $32.95 and $29.95 in the US).  This is, of
course, a specification for a different standard and so should be used
cautiously, but it will make it possible to become acquainted with some of
the main architectural features of 10646 and its repertoire while you're
waiting for your copy of the IS to show up.  

Unicode will also be holding an implementers workshop in Germany later this
year.  Contact the Unicode Consortium at unicode-inc@cairo.eng.sun.com for
more details.  Again, this is about Unicode and not 10646 but will almost
certainly include a discussion of the two standards and the relationship
between them.  

> - What else should a computer science student know about ISO 10646 
>   in the future?
> 

Why, everything, of course.  :-)


John H. Jenkins
John_Jenkins@taligent.com

#include <std_disclaimer.h>

In article <23302B@erik.naggum.no>, enag@ifi.uio.no (Erik Naggum) wrote:
> 
> Fritz Whittington <fritz@ra.csc.ti.com> writes:
> |
> |   I believe that the revised standard will come up for vote again
> |   around the end of the year.
> 
> This is incorrect.  Voting terminated on 1992-04-10, and ISO DIS 10646
> became ISO 10646, a fully official International Standard.  It has not
> been published and made available, yet that I know of.  Partly due to
> the size of the standard, partly due to the number of comments with
> votes, which I think have been reconciled by now (with the exception of
> Turkey's irrational request to replace ISO 8859-1 in the 00xx "row" with
> their favorite set, and Japan's more reasonable concerns).
> 

The vote actually ended at the end of May.  Some countries got their votes
in late and were not counted.   

I believe that the comments from Poland and Tunisia were not accomodated
for reasons similar to Turkey's (who wanted Latin 9 used instead of Latin 1
in the first 256 code points; nobody else would agree to that).  The UK and
Denmark were accomodated and appear to be satisfied with the accomodations.
 

Turkey, BTW, used its diplomatic corps to exert pressure on other countries
to go along with them.  I think this backfired and lost them sympathy
rather than winning them any.  

There was a lot of effort expended to try to satisfy Korea, but it's
unclear whether it actually did.  There is considerable disagreement within
Korea as to whether precomposed or combining methods should be used to
represent Hangul; the IS includes both, but only by putting several
thousand precomposed Hangul within the user space (unofficially, of
course). Nobody can tell if Korea will, in the end, find this satisfactory
or not.

One of Japan's reasons for voting no was procedural (they felt that it
violated ISO procedures to have a second DIS with so many fundamental
changes).  This could not be accomodated.  Some of their technical comments
were accomodated and some weren't.  For example, Japan's desire to have a
swap zone within 10646 large enough to hold all of JIS was generally felt
to be unreasonable, given the fact that everything in JIS is already
included in the CJK-JRG URO (which is part of 10646). 

BTW, I'm told that 10646 set a record for the number of countries
participating in the vote.  

John H. Jenkins
John_Jenkins@taligent.com

#include <std_disclaimer.h.>

I wrote:
||
||  Voting terminated on 1992-04-10...

John H. Jenkins <jenkinsj@blowfish.taligent.com> wrote:
|
|   The vote actually ended at the end of May.  Some countries got their
|   votes in late and were not counted.

Yes, that's right.  I confused the votes on ISO 10744 and ISO 10646.
1992-05-30 was the vote termination date for ISO DIS 10646.

|   I believe that the comments from Poland and Tunisia were not
|   accomodated for reasons similar to Turkey's (who wanted Latin 9 used
|   instead of Latin 1 in the first 256 code points; nobody else would
|   agree to that).  The UK and Denmark were accomodated and appear to
|   be satisfied with the accomodations.

Part of the reason Denmark voted NO was to allay the Turkish claims,
too.  This was a formally supported move, but several countries just
voted YES to indicate their displeasure.

|   Turkey, BTW, used its diplomatic corps to exert pressure on other
|   countries to go along with them.  I think this backfired and lost
|   them sympathy rather than winning them any.

Certainly!  Norway got into voting at the behest of Iceland, who were
worried about ISO 8859-1 being replaced, and their ETH and THORN going
the way of the dodo.  Turkey had completely neglected to allow for the
characters they suggested to be replaced, which also made people angry
at them.  I saw both letters, and found Turkey totally mindless.
Iceland's concern was understandable, but they did at least not use
their diplomatic corps to counter-act Turkey.

|   BTW, I'm told that 10646 set a record for the number of countries
|   participating in the vote.

Maybe we can "thank" Turkey for that... :-)

Best regards,
</Erik>
--
Erik Naggum             |  ISO  8879 SGML     |      +47 295 0313
                        |  ISO 10744 HyTime   |
<erik@naggum.no>        |  ISO 10646 UCS      |      Memento, terrigena.
<enag@ifi.uio.no>       |  ISO  9899 C        |      Memento, vita brevis.

What means UCS (after ISO 10646) in Erik Naggum's .sig? United Character
Set? What other terms are defined in ISO 10646 (MBP, ...)?

Markus

-- 
Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
-A distributed system is one in which the failure of a computer you didn't-
-even know existed can render your own computer unusable. (Leslie Lamport)-

Markus Kuhn <mskuhn@immd4.informatik.uni-erlangen.de> writes:
|
|   What means UCS (after ISO 10646) in Erik Naggum's .sig? United Character
|   Set? What other terms are defined in ISO 10646 (MBP, ...)?

That's the abbreviation after the standard's full name, which is (was,
at DIS stage, anyway):

	Information technology -- Universal Multiple-Octet Coded
	Character Set (UCS)

I seem to remember that somebody wanted "multiple-octet" removed from
the title.  Anybody else know the details on this?

Best regards,
</Erik>
--
Erik Naggum             |  ISO  8879 SGML     |      +47 295 0313
                        |  ISO 10744 HyTime   |
<erik@naggum.no>        |  ISO 10646 UCS      |      Memento, terrigena.
<enag@ifi.uio.no>       |  ISO  9899 C        |      Memento, vita brevis.

In article <23303A@erik.naggum.no>, enag@ifi.uio.no (Erik Naggum) wrote:
> 
> Markus Kuhn <mskuhn@immd4.informatik.uni-erlangen.de> writes:
> |
> |   What means UCS (after ISO 10646) in Erik Naggum's .sig? United Character
> |   Set? What other terms are defined in ISO 10646 (MBP, ...)?
> 
> That's the abbreviation after the standard's full name, which is (was,
> at DIS stage, anyway):
> 
> 	Information technology -- Universal Multiple-Octet Coded
> 	Character Set (UCS)
> 
> I seem to remember that somebody wanted "multiple-octet" removed from
> the title.  Anybody else know the details on this?
> 

I don't know if it came up at WG2 in Seoul, but it did come up at X3L2 last
month.  
I'd have to actually dig through my list of country comments to see if
anybody in fact suggested the name change (I believe somebody did), but I'm
too lazy to do that right now.  

The problem with the name "multiple-octet" is that it implies that 10646
text can be processed as a series of bytes instead of as a series of words
or longwords.  This can be disastrous in C because of the NUL octets in
some character codes.   Changing the name to avoid this implication would
be nice, but ISO regulations would consider changing the standard's name a
major revision, which would require a third ballot, and *nobody* wanted a
third ballot. 

John H. Jenkins
John_Jenkins@taligent.com

Even though John Jenkins and Erik Naggum have already responded to some
recent second-hand statements about 10646, I think it is worth emphasizing a
couple of points.

1. 10646 is now an official ISO standard which defines a character
   repertoire *aimed* at covering the written form of all languages.
   [Some languages cannot be represented yet since the scripts used
   to write them have yet to be incorporated into 10646; namely, the
   writing systems based on the Burmese, Ethiopic, Khmer, Mongolian,
   Sinhalese, and Tibetan scripts.  These are planned for a future
   version along with other archaic and less-used scripts. ]

2. 10646 defines two encodings for these characters:  a 32-bit canonical
   form and a 16-bit Base Multilingual Plane (BMP).  The 16-bit form, also
   called UCS2 (universal character set - 2 byte form), can be transformed
   into the 32-bit form (UCS4) by zero extending the 16-bit value to 32 bits.

3. No characters are currently defined outside of the BMP.  The BMP is
   derived from Unicode 1.0 with some changes; Unicode 1.1 will be identical
   to the BMP.  [In fact, Unicode 1.0, volume 2, which publishes the Han
   character encodings, is already identical to the Han encodings in the BMP;
   other changes required of Unicode 1.0, volume 1, in order to accomodate
   the new 10646 are also documented in volume 2.]

   The first 256 encodings of both the USC2 and UCS4 forms are identical to
   ISO8859-1 (of which ASCII constitutes the first 128 elements).

4. Software should not interpret 10646 encoded data as a sequence of bytes
   for the purpose of determining character values; only 16-bit and 32-bit
   unsigned integral units should be used.  [Byte swapping may be necessary
   to deal with mixed endian envrionments; 10646 does define a mechanism to
   ensure that the correct endian order can be determined].

5.  There are *no* holes in the encoding space of either UCS4 or UCS2 (BMP).
    There *are* unassigned code points (bit values); but these may be assigned
    in future versions of the standard.  [This differs from EUC, Shift Jis,
    and other similar encoding techniques that *do* have holes in their
    encoding spaces.]

6.  The language of the text encoded with 10646 is not determined by 10646;
    a higher level protocol such as an environment variable, an escape
    sequence, or other out-of-band data must supply the language which is
    being represented.

7.  10646 does not allow creating new *characters* by using "combining
    characters."  The semantics of combining characters is that their
    visual form is intended to "combine" with the visual form of a base
    character.  The combination so produced is not a "character" in the
    sense of "a member of a character set."  However, an application or
    user may choose to interpret the combination as a "letter" (i.e., a
    unit of writing) in the context of their written language.

8.  The issue of combining characters in 10646 is *very* different from
    the issue of variable length encodings in ShiftJis, EUC Multibyte,
    and other similar variable length encoding techniques.  In these cases,
    one *cannot* interpret a sequence of bytes *as a character in a
    character set* without scanning from the beginning of the string (or
    by synchronizing to a character encoding boundary).  In contrast, with
    10646, a program can randomly index (by 16-bit unit) any character
    encoding (code element) in a string and be assured of finding a proper
    character encoding; with Shift Jis, a random index (by byte), may land
    one right in the middle of a character encoding and thus yield garbage.

9.  Contrary to claims otherwise, 16-bits is adequate for representing all
    of the world's languages if two principles are followed:  (1) unify
    different uses of the same script and encode only once instance of
    a given script; and (2) allow characters to combine in a visual sense
    in order to produce more complex shapes or forms from simpler shapes or
    forms.  Both of these principles are followed by Unicode (and 10646 BMP).
    It is unlikely that the full 32-bit canonical forms will ever be needed
    except for perhaps compatibility or efficiency reasons.  However, in
    terms of actually representing all languages, 16-bits is quite adequate.

    [Personally, I will bet my Sparcstation that a number of major vendors,
    including Apple and Microsoft, will never support the 32-bit form for
    internal processing.  It simply isn't needed.]

Finally, I'd like to note that  an Implementers Workshop for Unicode &
10646 is being planned for early December in Frankfurt, Germany.  A formal
announcement will be forthcoming.

Glenn Adams