🚀 go-pugleaf

Forwarded message:
:Newsgroup  comp.software.international
:Subject  FAQ for wordlists
:Date issue 1 25/4/94
:Organisation  Computer Hyphenation Ltd

Here is an offering for the FAQ of comp.software.international on
wordlists, when someone gets round to compiling one.
IMHO dictionaries should be another section of the FAQ.

If you think I have made any mistakes, or have anything to add, please
contact me by email.

Please do not email me large amounts of text, as I have to pay for
bandwidth.

I will post this monthly until the FAQ is organised properly.

--------------------------------------------------------------------

WORDLISTS
*********
Lists of words in languages are very useful in many fields of
linguistic research, spelling error correctors and the like.

ftp sites
---------
Back in 1992 Jorge Stolfi then of DEC Systems Research Center, 130
Lytton Avenue, Palo Alto CA 94301 collected an excellent set of
wordlists in dutch, english, german, italian, norwegian, and swedish,
before moving on to other work. These are still available on
gatekeeper.dec.com/.8/misc/stolfi-wordlists
They are compressed using a special algorithm, decompression C source
code is provided, and then compressed even more.  These have
propagated around internet and may be found on many ftp sites.

If you "archie" for a language  you will find many wordlists labelled
say "German", and most will be Stolfi's.  As these lists are quite
large it is important that you try to identify these duplicates,
without getting the files.  Stolfi's readme file identifies them
uniquely if present, otherwise where they have been decompressed and
conventionally recompressed, the size of the file is a good indicator
of duplicates.

The best compilation of wordlists at the time of writing is now held
on black.ox.ac.uk\wordlists administered by Paul Leyland
<pcl@black.ox.ac.uk>.   His list of directories is at the time of
writing: american, aussie, chinese, computer, danish, dutch, french,
german, italian, japanese, latin, literature, movieTV, names,
norwegian, places, polish, random, religion, russian, science,
spanish, swedish, yiddish.  This list may not be complete, as things
are often added.  The major European languages are well covered with
long lists, but the remainder are the best available, some being quite
short.

If you:
have other wordlists available,
can find other wordlists on internet,
are willing to collect wordlists in any of the minor languages,
or improve on any already there.
Please contact Paul Leyland direct.

Compilation of wordlists
------------------------
If you cannot find what you want, via archie, or the instructions
above, you may well want to produce your own list, and then hopefully,
Please! put it in the public domain.  If so please offer it to Paul
Leyland when finished.  The following assumes that you are creating a
wordlist on a shoestring, or as a minor project without funding.  A
properly funded wordlist is called a corpus, and is a different
proposition.

A good source of information is to subscribe to soc.culture.???, or
any newsgroup which is appropriate, get the FAQ, and monitor the
newsgroup for at least a month.

Electronic Newspapers
.....................
The advent of electronic newspapers, and listservers on the net
provides an excellent opportunity to collect wordlists with minimal
effort.  The quality of writing is usually good, they contain short
articles on many subjects, by many people, and so are an excellent
source of words.  A disadvantage is that many are in an electronic, or
unaccented version of the language.

Newsgroups
..........
The newsgroups soc.culture.* etc. are also a source of wordlists.  The
writing and spelling are often of an appalling standard, and the text
is mostly unaccented.  Subjects are varied, but politics and religion
produce a very high proportion of postings, this produces highly
skewed lists, for instance in alt.culture.indonesia "allah" is in the
twenty most frequently used words.  I do not believe that this is true
for normal indonesian text, or newspapers.

There are about 6000 newsgroups on the net, but your server probably
only holds the 1500 common ones.  many others are available which
typically are used by a single university.  Ask you sysop for a list,
you may find something interesting.

Press offices
.............
Government Press offices etc. are now putting news on the net, and can
be used.  They however tend to concentrate on Political news, and thus
have a highly skewed, and small, selection of words, although the
quality of text is good.

Dictionaries
............
It may seem obvious, but dictionaries can be converted into wordlists.
Bilingual language1 to language2 dictionaries can with a bit of very!!
very!! nifty editing can be made into wordlists for either language1
or language2.  There is always a problem with dictionaries that they
contain words which are interesting to linguists, and not words which
are used in real text, or which people really speak.  Compound words
and derivative words are under represented.  They are however fully
accented and the words which they contain are correctly spelled.

Accented or unaccented?
.......................
If at all possible collect words with correctly accented and modified
roman characters i.e the slashed 'l' in polish, e acute in french.
The easiest words to find however are from internet, where it is
convention to use a computer version of the language which uses the
unaccented versions of characters.
Beware of the accented status of upper case characters, in some
languages, by convention,  lower case versions of a character are
accented, whereas upper case are often unaccented.

Non roman languages
...................
The vast majority of Russian, Bulgarian, Ukrainian, and Greek on
internet is transliterated into roman characters.  Only a little  is
in the correct characters.  Several vowels in these languages
transliterate into the same roman character, so you cannot get simply
back to the correct character set from transliterated text.  Wordlists
for non roman languages should be in the original character set, those
in transliterated versions are of not much use.

How many words?
...............
For the simple non compounding languages,  English, French, Italian
etc. about 50,000 to 100,000 words will cover almost everything used
in normal  text.  For the agglutinative languages where it is common
to add words to words to words, such as German, or Finnish,  Several
hundred thousand are required, and worse the actual compound words
used keep changing.  These are targets to aim for, wordlists of any
length are useful.

Copyright
.........
IMHO as a non lawyer, the copyright of a piece of ordinary writing
rests in the artistic effort in writing it.  This is totally lost when
you make a wordlist, so provided that you have the right to read it,
the copyright, of any wordlist produced, is yours and you can put it
in the public domain.  Dictionaries may be different in that the
artistic effort is arguably in the selection of the words themselves.

Readme file
...........
Each wordlist should have a readme file giving relevant information.
Please state who you are and your email address for queries, but to
reduce these the following should be included:

your name and email adress.
the source of your list.
any gross bias in the words found, political, religious, etc.
The character set being used, with the representation of each of the
accented characters defined.  Not everyone used TEX or your favourite
code page!  It is a real pain decoding the character set with a paper
dictionary!  You would be surprised how many conflicting code pages
and character sets there are, 7, 8, 16, 32, 64 bit, ISO, PC, Mac, Gem,
Word Perfect, Unicode, quite literally hundreds.
whether misspellings have been removed.
whether proper nouns are capitalised, capitalisation has been left as
found, or all words have been lowercased. IMHO proper nouns are best
capitalised if that is the convention in that language.

Number of occurrences
....................
If one looks at the seminal work of the Brown corpus, available from
the Oxford Text Archive. One will find that the number of occurrences
of that word, in the text examined, is carefully tabulated. This is
very useful in many fields, and its addition to any wordlist is a
distinct advantage.  If used for a spelling corrector, it is possible
to construct  several dictionaries, one of common words held in main
memory, less common words held on disk and so on.

Two small public domain ANSI C source code programs to facilitate
their collection, are available currently from Dave Fawthrop
<hyphen@ibmpcug.co.uk>, but hopefully they will shortly be on
black.ox.ac.uk.

These are "one_word", which takes ASCII text, splits it at white
space, strips of punctuation at both ends of the word, changes the
word to lower case if required, and prints out the words one per line.
Both the definition of lower case and the definition of punctuation
are simply adaptable to the character set in use.  The output from
this is then sorted with any gutsy sort.  "uniq_num" takes this sorted
output, and prints out each word followed by the number of times it
occurred.  The above sequence with edits as required can be repeated
ad infinitum, with the occurrences collected properly.
----------------------------------------------------------------------
Dave Fawthrop, <hyphen@ibmpcug.co.uk>.
-- God loved the World so much that he gave his only Son, so that  --
-- anyone who believes in (trusts, clings to, relies on) him shall --
-- not perish, but have eternal life. STARTING NOW! *IT'S GREAT*   --