🚀 go-pugleaf

Here are the consolidated responses to the original question from
Steve R. Billings (srb@world.std.com):

     I'm trying to get some measure of the performance costs incurred
     by converting back and forth between character sets.

     We'll be getting SJIS or J/EUC as input, converting
     to Unicode for internal processing, and storing in a database
     in SJIS or J/EUC.

     In case your curious, here's why: We're using Visix Galaxy
     (which supports Unicode) for internal processing support,
     and Oracle and Sybase as the databases. The workstation in
     question (HI/UX, Hitachi's Japanese variant of HP/UX)
     supports SJIS and J/EUC.

     The use of Unicode internally will greatly simplify development
     since Galaxy provides alot of functionality, but we're
     concerned about the performance hit incurred from the conversions.

     Anyone have any guidelines for estimating conversion timing?


Jeff
INSOFT-L List Manager

*========================================================================*
 Jeff Bowyer                         EMail:
 Computing Center		     jbowyer@cis.vutbr.cz
 Technical University of Brno
 Udolni 19, 602 00 BRNO
 Czech Republic
*========================================================================*


Sender: Chuck Houpt (chuck@lec.com)
        Language Engineering Corp.

Although I haven't run any timing tests, I'd guess that the conversion will
be linear on the number of characters.  You will probably find that you can
convert characters much faster than you can write them to disk.

Plan 9's tcs (translate character set) program uses a finite state machine
and a big lookup table to do JIS/Unicode conversion - this is probably
linear and could be optimized to be really fast.


Sender: Jon Wtte (d88-jwa@nada.kth.se)

Character conversion is not your major concern when you talk to
a database.

Why do I say that?
Because for the most part, you can do character set conversion using
a simple look-up table (or a chain of two dimensions, in the case
where you have mixed one- and two-byte characters) As long as you
have the memory (around 2-300K) for the tables, you should really
not have any problems performance-wise, as we are talking 10
instructions per character (and 2 of those being memory accesses)

Your user interface and disk I/O will outshine conversion by an
order of magnitude or more.


Sender: Martin J. Duerst <mduerst@ifi.unizh.ch>
        Institut fu"r Informatik
        der Universita"t Zu"rich

I don not have any exact figures, but expect it to be very!!! fast.
Basically, what you have to do is to look up a number in a table,
sort of like

short uni2euc [65536];
short euc2uni [65536];

short uni, euc;

for (/* each character */)  {
	uni = euc2uni[euc];	/* or vice versa */
}

As not all of Unicode are kanji, the tables can be shorter.
There is some additional stuff for euc resp. sjis because
you have to decide between kanji and ascii, and shift bytes
if they aren't alligned.

For a test, take a conversion tool between sjis and euc,
the times that you will get will not be far away from what
you can expect.

By the way, could you give more information about Visix Galaxy?

>Thanks,
>Steve Billings
>srb@world.std.com
Intresting that you have to ask such questions given your mail address.
Thought people at such an address would know more than others :-).