🚀 go-pugleaf

Hello!

Howard Kapustein wrote:
> 
> We're using curses on Solaris 8.
> Which appears to support EUC only.
> 
> int euc = wgetwch() gets a character of input and addwch(c) displays a
> character.
> 
> I need the data as Unicode.
> Specifically, UCS-2.
> [UCS-4 also works.]
> I'd even settle for UTF-8 (I've got UTF conversion routines).
> 
> How do I reliably transform EUC<->Unicode?
> 
> Locale settings don't help.
> Looks like Sun (others?) encode EUC from the keyboard as a 4-byte
> value where c & 0x30000000 == code set (see csetno() definition in
> Sun's header).
> 
> Only problem is, they don't say what code set 0, 1, 2 or 3 really is.
> Looks like CS1 == JIS-X-0208
> Yes?
> What about the rest?
> 
> And that's fine for input.
> For output, I've got a Unicode character like 0x30A1 (lower case
> katakana a) -- how do I make that a valid EUC character curses will
> recognize?
> 
> Help!
> 
>     - Howard

We ran into similar problem last year - our Application Server was
written in C (used EUC for Japanese on Solaris) and we wanted to pass
the data as Unicode to Java servlet (to our JSP layer).
It did not work and then we found why -
on a Web page (cannot find URL now) of Sun documentation where they said
that their C run-time library's
  wchar
is NOT Unicode, it's Sun's proprietary implementation.

That is, Java expected one value while our C code produced wchar
with abosolutely different value. We always use Japanese word 'test'
(te-su-to) because we wrote down the values of these 3 Japanese letters
and thus can check it during debugging:
EUC          A4C6      A4B9     A4C8
SJIS         82C4      82B7     82C6
UTF-8        E381A6    E38199   E381A8
UCS-2        3066      3059     3068        
(big-endian, it's what Java expects)

So in wchar* variable we expected to see the above UCS-2 values while
really we saw something completely different...

The solution was to have just EUC in our C code, don't do any conversion
in C/C++ code and let _Java_ do it (this works 100% OK):
 - we send EUC to Java (as a byte array), not Unicode
 - while creating Java String object from this byte array, we provide
   the encoding parameter - "EUC" - letting Java know what the source
   data encoding is.

If you don't like the above solution, then you can use a heavier one -
attach ICU library to your C/C++ code and let ICU do the conversion
from EUC to UCS-2.

Links:
1) Java's own encoding names to be used in Java conversions
  (for versions 1.2, 1.3 and 1.4):
http://java.sun.com/products/jdk/1.2/docs/guide/internat/encoding.doc.html
http://java.sun.com/j2se/1.3/docs/guide/intl/encoding.doc.html
http://java.sun.com/j2se/1.4/docs/guide/intl/encoding.doc.html

2) ICU library for C/C++:
http://oss.software.ibm.com/developerworks/opensource/icu/project/index.html


-- 
Regards,
Paul Gorodyansky
"Cyrillic (Russian): instructions for Windows and Internet": 
http://ourworld.compuserve.com/homepages/PaulGor/

Paul Gorodyansky wrote:
>...
> on a Web page (cannot find URL now) of Sun documentation where they said
> that their C run-time library's
>   wchar
> is NOT Unicode, it's Sun's proprietary implementation.


By the way, can anybody answer how people write Unicode-enabled
programs in C/C++ under Solaris if wchar there is not Unicode?

Solaris 8 claims to fully support Unicode, I cna even choose UTF-8
as a locale but what does it really means if C/C++ do NOT support
Unicode there???


-- 
Regards,
Paul Gorodyansky
"Cyrillic (Russian): instructions for Windows and Internet": 
http://ourworld.compuserve.com/homepages/PaulGor/

Paul Gorodyansky wrote:
> 
> Paul Gorodyansky wrote:
> >...
> > on a Web page (cannot find URL now) of Sun documentation where they said
> > that their C run-time library's
> >   wchar
> > is NOT Unicode, it's Sun's proprietary implementation.
> 
> By the way, can anybody answer how people write Unicode-enabled
> programs in C/C++ under Solaris if wchar there is not Unicode?
> 
> Solaris 8 claims to fully support Unicode, I cna even choose UTF-8
> as a locale but what does it really means if C/C++ do NOT support
> Unicode there???

I don't know - but I guess that if you set UTF-8 as your locales
character set - then it will correctly convert to / from UTF-8 as
a multi byte character set to Sun's (not unicode) wchar encoding.

Note also that the C standard allows the encoding of wide characters
to be locale specific. So in a UTF-8 locale it could be UNICODE and
in a Japanise locale it could be something completely different.

-- 
Stephen Baynes    CEng  MBCS                Stephen.Baynes@soton.sc.philips.com
Philips Semiconductors Ltd                  
Southampton SO15 0DJ                        +44 (0)23 80316431
United Kingdom                              My views are my own.

Stephen Baynes <stephen.baynes@soton.sc.philips.com> writes:

> the C standard allows the encoding of wide characters
> to be locale specific. So in a UTF-8 locale it could be UNICODE and
> in a Japanise locale it could be something completely different.

Quite right.

This is a somewhat controversial point.  I recall that at least one
member of the C Standard committee was surprised to hear about this
property of the standard.  It's conceivable that the next iteration of
the C standard will require ISO 10646 for wchar_t.  (By 2009, runtime
translation tables will be cheap, right?  :-)

Stephen Baynes wrote:
> 
> Paul Gorodyansky wrote:
> >
> > Paul Gorodyansky wrote:
> > >...
> > > on a Web page (cannot find URL now) of Sun documentation where they said
> > > that their C run-time library's
> > >   wchar
> > > is NOT Unicode, it's Sun's proprietary implementation.
> >
> > By the way, can anybody answer how people write Unicode-enabled
> > programs in C/C++ under Solaris if wchar there is not Unicode?
> >
> > Solaris 8 claims to fully support Unicode, I can even choose UTF-8
> > as a locale but what does it really means if C/C++ do NOT support
> > Unicode there???
> 
> I don't know - but I guess that if you set UTF-8 as your locales
> character set - then it will correctly convert to / from UTF-8 as
> a multi byte character set to Sun's (not unicode) wchar encoding.

Thanks! It makes sense. That is, it means that now we have UTF-8
_code page_. It's like MultiByteToWideChar() under Windows that 
converts non-Unicode to wchar_t - the code page must exist on the
computer for this function to work. For example, U.S. NT 4.0 that
does not have Japanese code page - that function can not be used
there to convert Japanese text to wchar.


> 
> Note also that the C standard allows the encoding of wide characters
> to be locale specific. So in a UTF-8 locale it could be UNICODE and
> in a Japanise locale it could be something completely different.

No, here I can not agree. wchar can NOT be locale-specific, otherwise
none of our programs would work :)

C standard may allow wchar to be _platform_ specific (i.e. Windows
has it as UCS-2 while Sun - as their own implementation), but
not Locale-specific, wchar has nothing to do with Locales:

for example, we have I18n-enabled C/C++ code - as it should be -
single executable for all languages, wchar is always
"_platform-specific_ wchar" despite the locale being it UTF-8 or French
or Japanese - locale serves absolutely different purpose -
decimal separators, sorting, etc. - it has nothing to do
with character representation.

So, in our current code - despite the locale - wchar under Sun is always
the same Sun's own form and under Windows it's UCS-2 - it does not matter
where our Sun or Windows NT computer is a Japanese or a French one.


-- 
Regards,
Paul Gorodyansky
"Cyrillic (Russian): instructions for Windows and Internet": 
http://ourworld.compuserve.com/homepages/PaulGor/

<3C22320B.E5838283@compuserve.com> divulged:

>So, in our current code - despite the locale - wchar under Sun is always
>the same Sun's own form and under Windows it's UCS-2 - it does not matter
>where our Sun or Windows NT computer is a Japanese or a French one.

in fact it is implementation defined.  which is to say that it's up to the
compiler vendor to decide what they want to do, e.g., if sun feels like
making it ucs-4 when compiling v9 code but a private bastardization
otherwise that is okay as far as the standard is concerned.

in fact the implementor could choose to do it one way in even numbered
threads (or statement-line number!) and another way in odd numbered ones,
so long as they insure that the proper mojo is applied so that it works
consistently, e.g., if thread #1 puts a latin small letter a with macron
(u+0101) into a wchar_t variable, which let's say the implementation stores
using ucs-2 encoding (0x00000101), and thread #2 puts the same letter into
a variable, which let's say is stored using the system (not program) locale
(which is, for the sake of this example, iso-8859-4) if possible otherwise
some sort of encoded value (which detail i will not go into, since it isn't
necessary) which in this case means 0x000000e0, then so long as an
evaluation of var1==var2 results in `true' the implementation works and
should be acceptable.  this would naturally be completely silly, but it
would be allowed, and it shouldn't matter to your code.  why shouldn't it
matter?  because you shouldn't compare a wchar_t to anything except a
wchar_t.

since i/o is usually locale dependent there is a potential for working
programs to fail, e.g., ``env LOCALE=ISO-8859-4 some_program | env
LOCALE=ISO-8859-1 another_program'' may not work as expected if the pipe
passes a u+0101.  (some_program outputs an 0xe0 which another_program
believes is a latin small a with grave.)

all that said, i believe that most implementations will migrate to a ucs or
utf oriented encoding of the storage.  ucs-2 and utf-8 both have strong
arguments -- pro and con -- so which is most likely still a little unclear.

-- 
okay, have a sig then

Hello!

those who know me have no need of my name wrote:
> 
> <3C22320B.E5838283@compuserve.com> divulged:
> 
> >So, in our current code - despite the locale - wchar under Sun is always
> >the same Sun's own form and under Windows it's UCS-2 - it does not matter
> >where our Sun or Windows NT computer is a Japanese or a French one.
> 
> in fact it is implementation defined.

Exactly.

>...  why shouldn't it matter?  because you shouldn't compare a wchar_t to
> anything except a wchar_t.

Exactly. I answered to this matter's question comparing wchar_t to
wchar_t and not anything else - the above message guessed that _wchar_t_
is locale-dependent, and I answered that it's not, it's implementation
dependent only.

> 
> since i/o is usually locale dependent there is a potential for working
> programs to fail, e.g., ``env LOCALE=ISO-8859-4 some_program | env
> LOCALE=ISO-8859-1 another_program'' may not work as expected if the pipe
> passes a u+0101.  (some_program outputs an 0xe0 which another_program
> believes is a latin small a with grave.)

No, we were talking _only_ about wchar_t. Again, based on our software
that works on Japanese, Polish (iso-8859-2), Latvian (iso-8859-4), etc.
we see that wchar_t under Sun is always the same Sun's own form.

> 
> all that said, i believe that most implementations will migrate to a ucs or
> utf oriented encoding of the storage.  ucs-2 and utf-8 both have strong
> arguments -- pro and con -- so which is most likely still a little unclear.

Unfortunately, I did not see any such plans for C compiler.

Also, we are using HP-UX and AIX and I guess they also have their own
implementation of wchar_t - we did not look though, because the fact that
I can not send wchar_t to Java code on our main Unix platform -
Solaris - was enough for us to cancel the original idea and we started
sending 'native', non Unicode data to Java...


-- 
Regards,
Paul Gorodyansky
"Cyrillic (Russian): instructions for Windows and Internet": 
http://ourworld.compuserve.com/homepages/PaulGor/