🚀 go-pugleaf

Here's the consolidated answers to the original question from Steve
Billings (steve@world.std.com):

     Is it true that Unicode cannot be mapped 1-to-1 to either S-JIS
     or EUC?

     In 14.5 hours I need to be able to talk intelligently on this subject
     before a very important audience.  All help is greatly appreciated!

[Note from Jeff: Several people responded immediately (I mean
                 IMMEDIATELY) and directly to Steve.  Great teamwork!
		 (and isn't email wonderful?) ]

Jeff
INSOFT-L List Manager

*========================================================================*
 Jeff Bowyer                         EMail:
 Computing Center		     jbowyer@cis.vutbr.cz
 Technical University of Brno
 Udolni 19, 602 00 BRNO
 Czech Republic
*========================================================================*


Sender: Glenn Adams <glenn@metis.com>
        Technical Director, Unicode Consortium

No, it is not true.  Unicode is a superset of both S-JIS and EUC-J!
No information whatsoever is lost in a round-trip conversion from
S-JIS or EUC-J to Unicode and back.  Unicode is also a superset of
the union of { ASCII, JIS X 0201, JIS X 0208, JIS X 0212 }.

If you wish to verify this for yourself, you can obtain the mapping
tables from JIS X 0201 and JIS X 0208 to Unicode by anonymous FTP on
METIS.COM in the directory /pub/csmaps.  It is trivial to construct
the S-JIS and EUC-J mappings from this data, S-JIS just being a
numerical transformation of this data, and EUC-J being the union of
ASCII, JIS X 0201 (8-bit right half), and JIS X 0208 with each code from
the latter ored with 0x8080.

Please inform your sources that their information is incorrect and
that they should check their facts.


Sender: Hiro Yoshioka <yoshioka@jrdv04.enet.dec.com>

It is not true.


Sender: Yang Li-qun <yang@nff.ncl.omron.co.jp>

First,
I think this question is a little unclear. Unicode and S-JIS
are encoding of certain character sets, but EUC is an encoding class/method.

You can say EUCJP, or EUCCH, for example, to mean EUC encoding of JIS
japanese character sets(JISX0201, JISX0208, JISX0212) or EUC encoding
of Chinese character set(GB2312).

Second, Unicode is a coded character set which contains all characters
used in Japaneese, Chinese and Korean, etc. But S-JIS only contains
Japanese character code.

So for every character set code in S-JIS, there is correspondent character
code in Unicode; but it is not true vice-versa.


Sender: Chiaki Ishikawa (pmcgw!personal-media.co.jp!ishikawa@uunet.UU.NET)

I don't think this is true. The design of Unicode aimed at achieving
round-trip conversion possible with the existing character code
standard such as JIS.

Now, I am not sure if this is all true for S-JIS or EUC, but given
S-JIS is a slight modification of JIS and EUC (assuming that we talk
here about EUC, Extended Unix Code for JIS, which hass again direct
mapping from JIS, I think that you can say there is one to one mapping
between Unicode and S-JIS and EUC.

HOWEVER, I am not an expert of Unicode. I wonder if Glenn is reading
this and cares to comment.


Sender: Martin J. Duerst <mduerst@ifi.unizh.ch>

Yes. There are a lot of characters defined in Unicode, starting from
Latin-1 (German umlaut, French accents,...) to phonetic, mathematical
symbols, Hebrew, Arabic, many Indian alphabets, Korean Hangul, and so on,
that cannot be found in S-JIS or EUC. So an arbitrary UNICODE text cannot
be mapped to S-JIS or EUC. This would not make sense, as S-JIS or EUC
just contain Kana, Kanji, and some Latin, Greek and Kyrillic.


BUT: If you start with S-JIS or EUC (or JIS, for that matter),
you CAN map a text 1-to-1 to UNICODE, and you can map that same text
back to S-JIS or EUC or JIS, and you will get EXACTLY the same text
you started with. This is called a round trip. And this applies to
many other national standards. So you cannot loose something if you
go to UNICODE, you can only gain. A round trip is also possible for
JIS-212, the Japanese extended Kanji character set.

Actual conversion is not algorithmical, as from S-JIS to EUC, for example,
but by table lookup. These tables are available by ftp, and their size
is no problem with the memory available in present-day computers. Also,
lookup-based conversion is in many cases faster than algorithmic conversion.


Sender: Gordon Vidaver (gordon@lec.com)

Here are the man pages for a utility called tcs that supposedly does just
what you ask:

TCS(1)                   USER COMMANDS                     TCS(1)

NAME
     tcs - translate character sets

SYNOPSIS
     tcs [ -slcv ] [ -f ics ] [ -t ocs ] [ file ... ]

DESCRIPTION
     Tcs interprets the named file(s) (standard input default) as
     a stream of characters from the ics character set or format,
     converts them to runes, and then converts them into a stream
     of  characters  from  the ocs character set or format on the
     standard output.  The default value for ics and ocs is  utf,
     the  UTF  encoding described in utf(6).  The -l option lists
     the character sets known to tcs.   Processing  continues  in
     the  face  of  conversion  errors  (the  -s  option prevents
     reporting of these errors).  The -c option forces the output
     to  contain  only correctly converted characters; otherwise,
     0x80 characters will be substituted for UTF encoding  errors
     and  0xFFFD  characters will substituted for unknown charac-
     ters.

     The -v  option  generates  various  diagnostic  and  summary
     information  on  standard error, or makes the -l output more
     verbose.

     Tcs recognizes an ever changing list of character sets.   In
     particular,  it  supports  a variety of Russian and Japanese
     encodings.  Some of the supported encodings are

     utf  (our) UTF encoding from X/Open

     utf1 UTF encoding from 10646

     ascii
          7-bit ASCII

     8859-1
          Latin-1 (Central European)

     8859-2
          Latin-2 (Czech .. Slovak)

     8859-3
          Latin-3 (Dutch .. Turkish)

     8859-4
          Latin-4 (Scandinavian)

     8859-5
          Part 5 (Cyrillic)

     8859-6
          Part 6 (Arabic)

     8859-7
          Part 7 (Greek)

     8859-8
          Part 8 (Hebrew)

     8859-9
          Latin-5 (Finnish .. Portuguese)

     koi8 KOI-8 KOI-8 (GOST 19769-74)

     jis  guesses between ISO 2022-JP, EUC or Shift-Jis

     gb   Chinese national standard (GB2312-80)

     big5 Big 5 (HKU version)

     unicode
          Unicode 1.0

     tis  Thai character set plus ASCII (TIS 620-1986)

     msdos
          IBM PC: CP 437

     atari
          ATARI-ST character set

EXAMPLES
     tcs -f 8859-1
          Convert 8859-1 (Latin-1) characters into UTF format.

     tcs -s -f jis
          Convert characters encoded in one of several shift  JIS
          encodings  into UTF format.  Unknown Kanji will be con-
          verted into 0xFFFD characters.

     tcs -lv
          Print an up to date list  of  the  supported  character
          sets.

SEE ALSO
     ascii(1), rune(2), utf(6).

A friend at my company did actually try to translate shift JIS into UNICODE
and had numerous complaints from tcs.  So whether in practice a conversion
is actually possible may still be an open question.  In theory, tcs should