🚀 go-pugleaf

In article <10003@plains.NoDak.edu> kkim@plains.NoDak.edu (kyongsok kim) writes:

   (Erik Naggum) writes:

   :Unicode is subject to endianism.

   could anyone please explain what "endianism" is?

Sorry for using an unwarrantedly technical term outside its original
domain.  Computers whose smallest addressable unit of information is
the octet (byte) need some ordering scheme for the octets to make up
units consisting of more than one octet, such as a 16-bit quantity, or
a 32-bit quantity.  There are basically two ways to do this, with
variations over the theme, called "big-endian" and "little-endian".

Big-endian octet order means that the "big end" (most significant
octet) comes first, and conversely for little-endian octet order.  By
way of example, consider the octet order for the 16-bit quantity
U+0040 (the commercial at-sign in Unicode).  A big-endian hardware
would represent this as

	+----+----+
	| 00 | 40 |
	+----+----+

(reading memory from low addresses at left to high addresses at
right), while a little-endian hardware would represent the same
numeric quantity or Unicode character as

	+----+----+
	| 40 | 00 |
	+----+----+

What I mean by "endianism", then, is the whole issue around the
portability of binary coded information when the order of larger-than-
octet units are moved around one octet at a time.  E.g. if a little-
endian machine writes a U+0040 to a file, it will be read as whatever
U+4000 is in Unicode on a big-endian machine, and exactly the same the
other way around.  It should be clear that interoperability will lose
significantly through this scheme, and if a choice is made, machines
who have made the other choice will hit a severe performance penalty.

   :Unicode employs floating diacritics for scripts which do not separate
   :the diacritic and the character to which it applies.

   most people in favor of iso 10646 attack floating diacritics.  how
   do floating diacritics and non-spacing characters (which i believe
   iso 10646 adopts) differ?  from end-users' point of view, these two
   seem one and the same.  am i missing something?

Consider the Norwegian and French words for a small restaurant,
spelled "cafe'" (where the ' serves as a floating acute accent for
rendering purposes in the absence of an international character set
standard in which we wouldn't need it :-).  In Norwegian, the acute
accent over e is optional, it's an ornament to indicate stress,
toneme, etc.  It's not orthographically required.  In French, an e
with acute is a different orthographic unit than plain, unadorned e.

This means that in Norwegian, we can make do with a floating acute
accent, since the function of the acute accent is to modify the
character with which is combined.  In French, however, they cannot
make do with a floating acute accent because the acute accent does not
have a function by itself.  Rather, the unit is "e with acute".

Then there's the Norwegian character "a with ring above", in which the
ring above has exactly the same nature as the acute accent in French.
If Norwegian was supposed to be written with "a*" (* substituting for
a non-existent non-spacing floating "ring above"), it would complicate
things for us to the point where we would have to vote a strong NO to
a standard forcing us to do this.  (Note that we can't vote against
Unicode, we can only "fail to adopt it".)

Of course, French and Norwegian are sufficiently important languages
that we've had all our characters represented in ISO 8859-1 (with the
possible exception of the French political faux pas with respect to
the "oe" ligature).  Some minority languages are less well off, to put
it mildly.  I've heard that East European languages employing a
heavily diacriticized Cyrillic script are suffering from the lack of
characters for their needs, and think that floating diacritics is the
answer to their problem.

So, to summarize, a diacritic mark may or may not be an integral part
of a character depending on orthographic conventions in the language
in question.  To treat a diacritic as floating when it is an integral
part of a character would be wrong, as would insisting on having all
possible combinations of a truly floating diacritic and the characters
with which it may be combined coded separately.

Now, ISO DIS 10646 is of the "insist on all combinations" persuasion,
but has non-spacing characters for languages in which the "separate
unit of information" is eminently the case (e.g. Hebrew).  I've come
to learn that this is overly restrictive in many, many cases.

Unicode allows a large number of floating diacritical marks in
languages which I don't have a shred of competence to make comments,
but several people have expressed the opinion that they're not really
floating for several languages.

Without a firm ruling in the standard or national standards on the
nature of the diacritical marks from orthographical conventions
employed, there is an annoying ambiguity between "cafe'" and "caf*" (*
now substituting for e with acute accent).  Is the * really an e plus
an ', or is a separate character, or is it vice versa?  As noted
above, the answer is different from French and Norwegian, although the
word is exactly the same!

The other problem with floating diacritics is that the number of
characters is not naturally bounded, a thought at which ISO
understandable shudders.  Unicode talks about bounding the displayable
number of characters (with diacritical marks) through extra-standard
means, while ISO wants do it with intra-standard means.  For instance,
a commercial at-sign with acute accent and cedilla below doesn't make
much sense.  What should a Unicode display device do with that
sequence of characters?

I am deeply indebted to Professor David Birnbaum for explaining this
to me in much detail, and I'm of course responsible for any mistakes.

Hope this has helped.

--
[Erik Naggum]           Professional Programmer        <enag@ifi.uio.no>
Naggum Software             Electronic Text          <erik@naggum.uu.no>
0118 OSLO, NORWAY       Computer Communications            +47-2-836-863

In article <ENAG.91May3200814@maud.ifi.uio.no> enag@ifi.uio.no (Erik Naggum) writes:
>Unicode allows a large number of floating diacritical marks in
>languages which I don't have a shred of competence to make comments,
>but several people have expressed the opinion that they're not really
>floating for several languages.

Yes, UNICODE does not care which language we are dealing with; note too
that one may have to combine characters from several sections of the
UNICODE in order to form a complete script.  The question then becomes: so
what?  If we insist on having diacritics that float for the languages that
have them possible and fixed for the languages that require them, someday
someone will type "e'" with a fixed diacritic while writing Norwegian, or a
floting in French, just to have something break for them.  As I understand
it, UNICODE only has non-floating diacritics for historical (compatibility)
reasons.  For example, "e'" is U+009E only for compatibility with Latin-1,
while the explicit coding is U+0065 U+0301.  I take it that at U+009E there
will just be an alias entry referring to U+0065 U+0301.

>The other problem with floating diacritics is that the number of
>characters is not naturally bounded, a thought at which ISO
>understandable shudders.  Unicode talks about bounding the displayable
>number of characters (with diacritical marks) through extra-standard
>means, while ISO wants do it with intra-standard means.  For instance,
>a commercial at-sign with acute accent and cedilla below doesn't make
>much sense.  What should a Unicode display device do with that
>sequence of characters?

In my opinion, it should take the @ sign and superimpose an acute accent
and tack a cedilla at the bottom.  A high-quality output device will
probably have a set of pre-finished combinations, but that doesn't prevent
it from using plain old superposition (or fancied-up superposition) as a
default solution.  After all, the combination tells it what it should look
like, right? 

Endianism is a tricky question, but in most cases there is precedent.  For
telecommunication, both CCITT and Internet standards advocate bigendianism
(Motorola style).  Check out what the sequence of bits are out of a
V.24/RS-232 port.  Bigendian.  Thus that is probably the preferred style
for interchange.  For word processors etc. there are usually
numeric fields which have had to be resolved; mostly as the style dominant
on the machine it was introduced on.
[P.S. As a programmer, I prefer littleendian (Intel) style; while a
bigendian hex dump is easier to read, littleendianism avoid many of the
problems with different variable sizes.   D.S.]

I also think there should be a recommended mangling scheme for converting
Unitext to ASCII text spectrum (NOT octet spectrum) for purpouses like
Internet mail, which not is very likely to change any time soon.  I have
given the question some thought but I am not going to say anything until I
have figured out a "safe" way that could also distinguish between Unitext
and ASCII text.

                                     /Peter
-- 
IDENTITY:   Anvin, H. Peter           STATUS:    Student
INTERNET:   hpa@casbah.acns.nwu.edu   FIDONET:   1:115/989.4
HAM RADIO:  N9ITP, SM4TKN             RBBSNET:   8:970/101.4
EDITOR OF:  The Stillwaters BBS List  TEACHING:  Swedish

In article <ENAG.91May3200814@maud.ifi.uio.no> enag@ifi.uio.no (Erik Naggum) writes:
>In article <10003@plains.NoDak.edu> kkim@plains.NoDak.edu (kyongsok kim) writes:
>>   could anyone please explain what "endianism" is?
>
>[Explaination of endianism deleted.]

I will just add that the terms "big endian" and "little endian"
probably originated as references the Gulliver's Travels, by Jonathan
Swift.  The Liliputians and Blefescuans (right?  I forget the names,
exactly) were having a war over whether one should open one's eggs at
the big end or the little end.  The futility of such an argument makes
a good analogy to the futility of trying to decide which byte ordering
scheme is better.

Jim

Spoken: Jim Williams    Domain: williams@nssdcs.gsfc.nasa.gov
Phone: +1 301 286-1131  UUCP:   uunet!mimsy!williams
USPS: NASA/GSFC, Code 933, Greenbelt, MD 20771
Motto: There is no 'd' in "kluge"!  It rhymes with "huge", not "sludge".

In article <ENAG.91May3200814@maud.ifi.uio.no> enag@ifi.uio.no
(Erik Naggum) writes:

>Consider the Norwegian and French words for a small restaurant,
>spelled "cafe'" (where the ' serves as a floating acute accent for
>rendering purposes in the absence of an international character set
>standard in which we wouldn't need it :-).  In Norwegian, the acute
>accent over e is optional, it's an ornament to indicate stress,
>toneme, etc.  It's not orthographically required.  In French, an e
>with acute is a different orthographic unit than plain, unadorned e.
>
>This means that in Norwegian, we can make do with a floating acute
>accent, since the function of the acute accent is to modify the
>character with which is combined.  In French, however, they cannot
>make do with a floating acute accent because the acute accent does not
>have a function by itself.  Rather, the unit is "e with acute".

I confess that I don't understand the problem.  Regardless of the
attributes of the underlying language, is there some reason why I
should care whether a character-diacritic combination is stored as
one code or two as long as (a) its image is properly rendered when
I need to look at it and (b) a program which consumes a text stream
that includes such (character-diacritic) combinations can
unambiguously determine its content?  (Of course, if I can meet
requirement "b" presumably I can meet requirement "a" as well.)

-- 
Chris Kern     ck@voa3.voa.gov     ...uunet!voa3!ck     +1 202-619-2020

In article <1991May4.180549.29162@voa3.VOA.GOV> ck@voa3.VOA.GOV 
(Chris Kern) writes:

>I confess that I don't understand the problem.  Regardless of the
>attributes of the underlying language, is there some reason why I
>should care whether a character-diacritic combination is stored as
>one code or two as long as (a) its image is properly rendered when
>I need to look at it and (b) a program which consumes a text stream
>that includes such (character-diacritic) combinations can
>unambiguously determine its content?

Yes and no.  One could encode English logographically, but we don't
do it because (among other things) people don't process English text
logographically; they do it by character.  Similarly, we can encode
Hebrew consonant plus vertically aligned vowel points and cantilla-
tion marks as single characters, but people don't work with Hebrew
text this way.

One practical consequence of encoding vowel+accentual_diacritic
variously is the way it affects natural classes.  I can search for
all words with long rising accents in Serbocroatian (graphically
an acute) more easily if the acute is a separate character.  I
would not want to conduct such a search in French, where letters
with acute do not constitute a natural class (i.e., where "acute"
does not have an independent meaning), but this is not an unnatural
type of search to make in Serbocroatian, comparable to searching for
all words with any other letter.

As another example, I can
strip the (orthographically optional) accents from a Serbocroatian
text more efficiently by searching and deleting the five accentual
diacritics than by searching and replacing each accented vowel by
its unaccented counterpart.  Again, this is not something one would
normally want to do for French.

One other issue is efficient use of character cells.  If there is
a small number of vowels and a small number of accent marks (to
use a common example and imprecise terminology), there isn't a lot
at stake.  But take a system with lots of vowel letters, lots of
accent marks, the possibility of multiple accent marks on a single
vowel, and you're talking about a lot of character cells if each 
one is to be treated as an indivisible unit.  And it is writing
systems where accent marks are productive units that are combined
ad hoc with a natural class of letters (such as vowels) that have
this large number of combinations.

At a certain level, the answer to your question is that it doesn't
matter.  This seems to be why 10646 and Unicode have been able to
take opposing positions on the issue; both are concerned with form,
rather than function, and anything that arrives at the correct form
fulfills the minimal requirements.  But there are plenty of writing
systems that aren't like English or like French and where you can
only support the full inventory of complex combinations either by
storing the combination as a sequence or by dedicating an extremely
large number of character cells.  For orthographies like this, the
former is more efficient and corresponds more directly to the types
of operations that users may want to perform on the text.

--David

=======================================================================
Professor David J. Birnbaum         djbpitt@vms.cis.pitt.edu [Internet]
The Royal York Apartments, #802     djbpitt@pittvms.bitnet   [Bitnet]
3955 Bigelow Boulevard              voice: 1-412-687-4653
Pittsburgh, PA  15123  USA          fax:   1-412-624-9714

In article <ENAG.91May3200814@maud.ifi.uio.no> enag@ifi.uio.no (Erik Naggum) writes:
> a commercial at-sign with acute accent and cedilla below doesn't make
> much sense.  What should a Unicode display device do with that
> sequence of characters?

                        ,
It better display it as @ (more or less), because someone's gonna use it.
                        '
If you don't believe that, then consider the use of "!#%^&*|" in C.
-- 
Peter da Silva.  `-_-'  peter@ferranti.com
+1 713 274 5180.  'U`  "Have you hugged your wolf today?"