🚀 go-pugleaf

THE FONT ENCODING MESS:
=======================

Definition:  encoding vector --- a mapping from numeric character 
codes to characters.  Characters may be referred to by character names.

Definition:  unencoded character --- a character in a font that is
not listed in the currently used encoding vector, hence not accessible.

FLAME ON

If you always work on the same platform, and always use the same
application, you may not yet have stumbled in the encoding morass.
And you may believe that encoding problems are something that
afflicts people working on `inferior' platforms (i.e. anything other
than what you are currently working on), with `inferior' applications.

On the other hand, if you ever move documents between platforms,
and often even just between applications on the same platform, you
may have run into problems, particularly with characters not in the
basic ASCII set, such as ligatures and accented characters. And sometimes
even with characters other than letters and digits in the ASCII set!

What gives?  Well, there is no universally agreed upon encoding
vector, and, given that we presently use 8-bit bytes to represent
characters, there cannot be such a thing, since there are many more
than 256 characters --- even just counting characters in Western languages.

Many encoding schemes agree on at least part of the code range,
namely the ASCII character set between 32 and 127.  But even there
one finds exceptions and variations (such as `grave' versus 
`quoteleft', `hyphen' versus `minus' etc).  So actually the only
really dependably constant part are the letters a-z, A-Z and the
digits 0-9.  Well, that is if we ignore EBCDIC... 

There are probably around 50 commonly used coding schemes in use,
each claiming to be general and important.  Some try to force
acceptance by having names that suggest standardization.  For
example, there is `standard roman encoding', which is `standard', 
but only on the MacIntosh, and there is MS Windows ANSI, which suggests a
connection with the American National Standards Institute, yet
changes when a new release of Windows comes out.  And the `native'
encoding of Adobe Type 1 text fonts is usually `Adobe StandardEncoding'.
On some other platforms support for scalable outline fonts is as yet
so weak that it isn't clear where they stand on the encoding issue.

All of these encoding schemes have drawbacks.  Adobe StandardEncoding
does not provide access to accented characters.  MS Windows ANSI does
not provide access to the ligatures `fi' and `fl'.  Mac `standard roman
encoding' does not provide access to some `common' accented characters 
(e.g. yacute, Yacute, scaron, Scaron, zcaron, Zcaron) or to the
icelandic characters (eth, Eth, thorn, Thorn).  And so on.
Hence it is not a question of choosing one of these and making it the
universal standard.

TeX alone has ten different encoding schemes for the code range 0 -
127, and one has to depend on agreed upon conventions about which
font uses which, since TeX has no provision for specifying an
encoding vector in its metric files (aside from an optional field
that may be used to record a short name for the encoding).  Did you
know that cmr5 uses a different encoding vector from cmr7 - which
supposedly is the same font just at another size?  And that none of
the encodings used by TeX agree with ASCII even in the restricted 64
- 127 range?

PostScript has the right idea in requiring an encoding vector in a
font, and in labelling glyphs with character names, not numbers.
Unfortunately few applications make it possible to exploit this.
Indeed, most use a fixed encoding scheme that the user cannot modify.

One would think that font generation software would be particularly
likely to take advantage of the notion of an encoding vector.  Yet
many of them use a hard-wired relationship between numbers and
character names.  That is, the character names are used merely as
aliases for numbers (NUL is 0, Eth is 1, eth is 2, Lslash is 3 etc)!
The character names in this case bear no relationship to the glyphs.
Just because something is called `Lslash' doesn't mean it is the
Polish letter `l' with a short inclined stroke through it.  Or just
because a character has the name `less' doesn't mean it represents
the mathematical relationship `less than'.  It could very well be
`exclamdown'.  This is the worst corruption of the PostScript
encoding vector idea! 

What makes the encoding problem particularly aggrevating is that only
rarely does software provide for reencoding - based on a user specified
encoding vector.  In MS Windows, for example, a font can be set up in
only two ways: (a) use the fonts `native' or `raw' encoding, or (b)
reencode the font to Windows ANSI.  And this decision is hard-wired
into the font metric file (PFM).  Reencoding a font on-the-fly is
unheard of.  The same goes for the Macintosh, where the FOND resource
specifies whether a font is to be used `uncoordinated' as opposed to
reencoded to Mac `standard roman encoding'.  The situation on some
other platforms is even worse.

Note that reencoding is more general than permuting the encoding
vector.  For example, TeX's virtual font mechanism allows one to
renumber characters, but it cannot make unencoded characters
accessible.  Correspondingly, one cannot `reencode' a PFM or TFM
file, since these metric file formats contain no information on unencoded
characters, and reencoding can make previously unencoded characters
accessible.  One has to refer back to more complete information, 
such as that provided in Adobe Font Metric (AFM) files.

Font metric files should be organized around characters, not numeric
character codes.  Note that PFM and TFM metric files do not provide
for specification of an encoding vector.  The FOND resource in a
Mac screen font does, but in a limited way (usually only noting the difference
between the fonts `native' encoding and Macintosh `standard roman encoding').
Adobe Font Metric (AFM) files, on the other hand, are organized
around character names (while also giving the encoding of the
`native' or `raw' form of the font).

When moving text files between platforms, there seem to be two
choices: (a) accept that applications on the two platforms used
different encoding schemes and provide a `translation filter',
or (b) carry around the information about what encoding is in use 
with the fonts or with the document.  

Presently we mostly depend on (a) above, which has the serious
drawback that characters that exists in the encoding used on one
platform but not the other are lost (16 of the 217 characters in MS
Windows ANSI do not exist in Mac `standard roman encoding', while 23
of the 223 characters in Mac `standard roman encoding' do not exist
in MS Windows ANSI).  Obviously the second alternative is better, but
there is precious little software out there now that supports this.

Conclusion:
===========

(1) We need to raise the level of awareness of the font encoding issue.

(2) We need to reduce platform bigotry.  This is NOT a Macintosh, or
a MS Windows, or a Unix problem - it is universal.  It is not a
problem specific to a particular word-processing application or to
TeX - it applies to all software dealing with text.

(3) We must accept that there can be no universally accepted encoding
(using 8-bit character codes) that is suitable for all purposes.

(4) When referring to a font, it is NOT sufficient to specify a font
name, one should say also what encoding is in effect.

(5) Documentation of software that enforces its own encoding vector
should clearly state this fact and spell out the encoding vector used.
Hidding such reencoding under the rug eventually leads to total confusion.

(6) Font metric information should be provided in a form that ties
metrics to characters, not to numeric character codes.  Or, if it
does use numeric codes, then it must make provision for specification
of an encoding vector mapping these numbers into character names.

(7) If a metric file format does not provide for specification of an
encoding vector (such as PFM and TFM), then at least there must be
easily accessible tools for making such metric files based on
arbitrary user selectable encoding vectors from metric file formats
that are complete (such as AFM).

(8) Applications (and operating systems) should provide for `on the
fly' reencoding of fonts.

Finally:  Let's hope that UNICODE with its 16-bit characters will help
resolve some of these problems.  Although, UNICODE is dead in the
water if it does not provide access to ligatures.  For there can be
no fine typesetting without ligatures.

FLAME OFF

Berthold K.P. Horn
Cambridge, Massachusetts, USA