🚀 go-pugleaf

Efraim Gotlib wrote:
> 
> What is the best way to compress long strings

It depends:

How long is long?
What do the strings contain?
How fast must compression/decompression be?
Any restrictions on the compressed string?

If the strings are mainly containing English text, I can imagine the
following encoding to work (however, I haven't made experiments):

Use a 5 bit code like that:

0 = space
1 = a
2 = b
...
26 = z
27 = .
28 = ,
29 = ;
30 = Uppercase follows
31 = Full 8 bit characters follow

Then use the following rule:

(1) Initially you are in 5 bit mode (since probably you start with a
letter).

(2) As long as you are in 5 bit mode:
(2a) As long as the character is one of the 5-bit encoded characters,
add
     the 5 bit code of that character, and remain in 5 bit mode.
(2b) If the character is an uppercase letter, add the Uppercase 5 bit
     code (31) and then the corresponding lowercase letter (since
uppercase
     letters are the exception, this is usually shorter than using
     6 bit encoding with uppercase letters). Keep 5 bit mode.
(2c) If the letter is none of the above, you add the 5 bit code
     "to 8 bit mode" (31), and re-process that character in 8 bit mode
     (see step 3)

(3) As long as you are in 8 bit mode:
(3a) As long as the characters are not in the 5 bit mode (or uppercase
     equivalents), add the usual ASCII code and procede in 8 bit mode.
(3b) Otherwise add the 8-bit-code as well, but then switch to 5 bit mode

(4) The bits produced this way must be grouped into 8 bit bytes, since
    this is the basic addressing unit. If the last byte happens to have
    five or more bits free, while being in 5 bit mode, just add a
    (31) 5 bit code - this doesn't add another character, but fills up
    space that you must allocate anyway, and saves you from storing
    a bit count instead of a byte count. The result can f.ex. be stored
    in a TP string as well, using the normal length byte (which
    obviously then gives the compressed size).

Example (63 letters):

Compressing this should give a compression to about 70% - nice.

This gives (5 bit codes are in "", 8 bit codes in '', 5 bit special
codes are in () ):

(upcase) "Compressing this should give a compression to about "
(8-bit) '70% ' (8-bit) '- ' "nice."

This gives 54*5 + 4*8 + 1*5 + 2*8 + 5*5 = 348 Bits = 43 Bytes and 4
bits.
Since only one bit is left to a full byte, adding an (8-bit) is not
necessary. The compressed string has a length of 44 bytes. Thats
69.8% of the original size.

This could still be improved; for example, you could make "uppercase"
letters for space, '.', ',', ';', giving you four more characters
in 5 bit mode (which actually have 10 bytes; useful especially
for characters which tend to be embedded between directly supported
characters. Also, since digits tend to group as well (together with
some usual characters like '-' and decimal point), you could make
an additional 4 bit code for them (would cost another 5 bit char for
switch, however), containing f.ex. '0' to '9', '+', '-', '.', ',',
space,
(5-bit), (8-bit). This would make your logic more complex (since
you'd have to switch between three modes), but *might* save a few
additional bytes (only experiments tell you if it does; the decreased
number of symbols in 5 bit mode might eat up the advantage of 4 bit
mode, esp. if you have very few numbers in your string).

However, maybe you have a look at comp.compression - they surely know
much better methods...