🚀 go-pugleaf

Here are more consolidated responses for the on-going discussion about
Soundex.


Jeff
INSOFT-L List Manager

*========================================================================*
 Jeff Bowyer                         EMail:
 Computing Center		     jbowyer@cis.vutbr.cz
 Technical University of Brno
 Udolni 19, 602 00 BRNO
 Czech Republic
*========================================================================*


Sender: "Kevin C. Gross" <0004056890@mcimail.com>

Jon Wtte (d88-jwa@nada.kth.se) wrote...

>Soundex doesn't even know about various european characters such
>as ao, ae, oe and ue, so stating "Soundex should work moderately
>to exceptionally well for most european languages" is plain wrong.
>You can do better for Swedish using a homegrown hack than the
>published Soundex algorithm (there was some discussion on this
>in swe.general; I think; a while ago. In Swedish of course...)

Uh, good morning, Jon.  The soundex algorithm totally ignores all vowels,
so the introduction of marked or shaped vowels and ligatures by other
languages using the Latin alphabet is really irrelevant.  It is
important that an algorithm take into consideration non-English consonants
like thorn and eth, which I have to date thrown into the closest bins
possible in my own implementation.

Kevin


Sender: Olle Jarnefors <ojarnef@admin.kth.se>
        Royal Institute of Technology, Stockholm

Jon Wtte (should have been W<a-diaeresis>tte) <d88-jwa@nada.kth.se> wrote:

> An INSOFT-L subscriber wrote:
> > I used soundex codes several years ago in one of my applications and I
> > think it would work moderately well for other European languages, or
> > transliterated text for or words from elsewhere.
> > - - -
> > ... There's more to it than a simple substitution, but for this
> > question it should suffice to say that within broad areas, soundex should
> > work moderately to exceptionally well in any of the Romance languages.
> > Since pronounciations don't really vary that much from language to language,
> > and since the order of the alphabet stays the same, it should work.
> >
> > This is of course, just theory.  I've never tried Soundex on more than a
> > couple of non-English words.  I would be interested to see how it goes.
>
> Soundex doesn't even know about various european characters such
> as ao, ae, oe and ue, so stating "Soundex should work moderately
> to exceptionally well for most european languages" is plain wrong.
> You can do better for Swedish using a homegrown hack than the
> published Soundex algorithm (there was some discussion on this
> in swe.general; I think; a while ago. In Swedish of course...)

Jon is right. The discussion took place in the news group
swnet.general last autumn. I summarize in English my own
contribution -- which wasn't contradicted -- here.

A concise description of Soundex and its history can be found in
Donald Knuth: The Art of Computer Programming. Vol. 3: Searching
and Sorting, p. 391-392. Addison-Wesley, 1973.

Soundex was patented as early as 1918. Its purpose is to
transform English last names to a short code (letter plus three
digits) which tends to give all spelling variants of the same
name the same code. The transformation was intentionally made
very simple and coarse to allow people working with large manual
lists or data bases to mentally calculate the Soundex code while
entering or searching for data.

The Soundex transformation consists of
-- keeping the first letter
-- disregarding all vowels, h and w
-- replacing consonants by 1-6. in the groups bfpv, cgjkqsxz, dt,
   l, mn, and r by digits 1-6 (while disregarding any directly
   following letter of the same group)
-- truncating the code after three digits or, for short codes,
   add trailing zeros.
For example Macdonald and McDonald but also Mostyn have the same
Soundex code, M235. An example of a Soundex "failure" is that
different codes are given to Rogers (R262) and Rodgers (R326).

Two conclusions can be drawn from this:
1) The effectiveness of Soundex is very dependent on the language
   of the treated names.
2) Even for the English language much better results should be
   possible with a more sophisticated algorithm (made feasible by
   present-day computers).

I'm not competent to devise a good algorithm for English but I
can here describe a fairly good algorithm for Swedish, to show at
least what level of complexity is needed in one localization
case.

Two factors determine the algorithm complexity for a certain
language: firstly how well the spelling of words in general
reflects their normal pronunciation, secondly the degree to which
spelling variants for personal names are accepted. Swedish has a
much better correspondance between pronunciation and spelling
than English (but not as good as German or Finnish). Compared to
many other languages Swedish, however, seems to be much more
liberal as regards name spelling variants. The very common last
name Eriksson for instance, can be spelt in at least 30 different
ways.

The following algorithm can be used to generate the primary
sorting key for large alphabetical lists of persons. In personal
name searching it can then be used to get a suitable form of the
search key for comparison with the stored data. As an example it
will normalize different forms of Eriksson such as Ericson,
Ericksohn, Erixon, and Erichszon to "erikson".

Algorithm for normalizing Swedish words according to pronunciation
------------------------------------------------------------------

Swedish uses the vowels   aeiouy<a*><a"><o">
and the consonants        bcdfghjklmnpqrstvwxz
(<a*>=a with ring, <a">=a with diaeresis, <o">=o with diaeresis)

Step 1: Replace capital letters by small letters.

Step 2: Apply these transformations in the given order:
   a) chr -> kr
   b) ch  -> ch (otherwise)
   c) ck  -> k
   d) c   -> k (otherwise if before one of aou<a*> or consonant or final)
   e) c   -> s (otherwise)
   f) f   -> f (before vowel)
   g) f   -> v (otherwise)
   h) th  -> t
   i) hj  -> hj
   j) h   -> h (before vowel)
   k) h   -> <nothing> (otherwise)
   l) ph  -> f
   m) q   -> k
   n) w -> v
   o) x   -> ks
   p) <a">r    -> <a">r
   q) <a">y    -> <a">y
   r) <a"><a"> -> <a"><a">
   s) <a">     -> e (otherwise)
   t) <hyphen> -> <space>

Step 3 Replace sequences of the same consonant by one consonant,
but keep sequences of the same vowel.