🚀 go-pugleaf

                        WORDLISTS FAQ
                        *************

               Created 19/4/97. Last modified 7/5/2000.
               URL http://www.hyphenologist.co.uk

Contents
********

     Introduction
     ftp sites
          The Moby Project
     Compilation of wordlists
          World Wide Web, WWW
          Etext
          Electronic Newspapers
          Listservers
          Newsgroups
          Paper Newspapers
          'Zines
          Academic papers
          Press offices
          Dictionaries
          Things to edit out of text
          Accented or unaccented?
          Non roman languages
          How many words?
          Copyright
          Readme file
          Number of occurrences
     Computer Hyphenation Home Page

Introduction
************

Here is an offering for the FAQ on WORDLISTS, for *.text.*
soc.culture.* *.lang*, *.doc.* newsgroups, or anywhere else that
seems reasonable where I post on about a 3 monthly cycle.

If you think I have made any mistakes, or have anything to add, please
contact Dave Fawthrop. by email <hyphen@c-h.win-uk.net>.

If you use this drop me an email, it may encourage me to spend money
and time on a rather crazy hobby.

(C) Copyright Public Domain, use at will :-)

Lists of words in languages are very useful in many fields of
linguistic research, spelling error correctors and the like.

ftp sites
=========

Back in 1992 Jorge Stolfi then of DEC Systems Research Center, 130
Lytton Avenue, Palo Alto CA 94301 collected an excellent set of
wordlists in dutch, english, german, italian, norwegian, and swedish,
before moving on to other work.  These are still available on
ftp://ftp.digital.com/pub/misc/stolfi-wordlists/ or
http://gatekeeper.dec.com/pub/misc/stolfi-wordlists/ They are
compressed using a special algorithm, decompression C source code is
provided, and then compressed even more. These have propagated around
internet and may be found on many ftp sites.

If you "archie" for a language you will find many wordlists labeled
say "German", and most will be Stolfi's. As these lists are quite
large it is important that you try to identify these duplicates,
without getting the files. Stolfi's readme file identifies them
uniquely if present, otherwise where they have been decompressed and
conventionally recompressed, the size of the file is a good indicator
of duplicates.

The best compilation of wordlists at the time of writing is on
ftp://ftp.ox.ac.uk/pub/wordlists/* (alias for sable.ox.ac.uk)
administered by Bob Douglas at bob@ftp.ox.ac.uk or alternatively
root@sable.ox.ac.uk will be able to help. The list of directories is
at the time of writing: afrikaans, american, aussie, chinese,
computer, croatian, czech, danish, databases, dictionaries, dutch,
esperanto, finnish, french, german, hindi, hungarian, italian,
japanese, latin, literature, movieTV, music, names, net, norwegian,
places, polish, random, religion, russian, science, spanish, swahili,
swedish, turkish, yiddish. This list may not be complete, as things
are often added. The major European languages are well covered with
long lists, but the remainder are the best available, some being
quite short, and some statistically suspect.

If you:

     have other wordlists available,
     can find other wordlists on internet,
     are willing to collect wordlists in any of the minor languages,
     or improve on any already there.

administered by Bob Douglas at bob@ftp.ox.ac.uk or alternatively
Please contact  direct.

The Moby Project
=================

In June 1996 Grady Ward announced that the fruits of the Moby project
were being placed in the public domain. The files comprising the
project are now available available from
ftp://ftp.dcs.shef.ac.uk/share/ilash/Moby/ftp://ftp.ox.ac.uk/pub/wordl
ists/Moby/ either as a complete distribution [26MB] or as set of
subprojects:

     Moby Hyphenator (American) 185,000 entries fully hyphenated
     mhyph.tar.Z  [980kB]
     Moby Language Word lists in five of the world's great languages
     mlang.tar.Z  [2.3MB]
     Moby Part-of-Speech (American) 230,000 entries fully described
     by part(s) of speech, listed in priority order mpos.tar.Z [1.2MB]
     Moby Pronunciator (American) 175,000 entries fully International
     Phonetic Alphabet coded mpron.tar.Z [3.1MB]
     Moby Shakespeare The complete unabridged works of Shakespeare
     mshak.tar.Z [2.3MB]
     Moby Thesaurus (American) 30,000 root words, 2.5 million synonyms and
     related words mthes.tar.Z [12MB]
     Moby Words (American) 610,000+ words and phrases. The largest word
     list in the world (American) mwords.tar.Z [4.0MB].

For more details, see: http://www.dcs.shef.ac.uk/research/ilash/Moby/

Compilation of wordlists
************************

If you cannot find what you want, via archie, WWW, or the instructions
above, you may well want to produce your own list, and then
hopefully, Please! put it in the public domain. If so please offer it
administered by Bob Douglas at bob@ftp.ox.ac.uk or alternatively
to  when finished. The following assumes that you are
creating a wordlist on a shoestring, or as a minor project without
funding. A properly funded wordlist is called a corpus, and is a
different proposition.

A good source of information is to subscribe to soc.culture.???, or
any newsgroup which is appropriate, get the FAQ, and monitor the
newsgroup for at least a month or two.

World Wide Web, WWW
===================

An excellent starting place, to search for information on any language
is the Home page for that country, which often contain pointers to
linguistic data, newspapers in the language, Government news services
etc. etc. These are available for the smallest countries and are
often run by expatriates working at American universities. To find
these home pages go to one of the WWW search engines: Lycos at
http://www.lycos.com, Webcrawler at http://www.webcrawler.com. Alta
Vista at http://altavista.com and search for 'languagename language'.
Otherwise go to Yahoo http://www.yahoo.com and follow the pointers.

Other worthwhile linguistics web sites to visit are:

Brian Kelk's wordlist pointers with information on word frequency and
letter frequency.http://www.cl.cam.ac.uk/users/bck1/menu.html Also
for phonetic alphabets (Alpha Bravo) and Hendrik Biebeler's
compilation of taught languages

A Multilingual lexicography web page at
http://egnatia.ee.auth.gr/~yhat/yiannis.

Links to many foreign language resources can be found at:
http://www.cc.utah.edu/~coj6886/jltc.html
Sorted List of the Most Popular Languages
Lots of linguistic stuff with links:
Yamada Language Center http://babel.uoregon.edu/yamada.html
Foreign Language Learning Resources - CALL http://www.call.gov/index.htm
The Human-Languages Page http://rucus.ru.ac.za/~grahams/hl.html
Multilingual Software Digest http://www.gy.com/www/mb.htm

Etext
=====

There is a mass of electronic text out there but mostly in
English/American. If it is important and out of copyright someone is
trying to put it on the WWW. I particularly recommend the worlds best
etext site Internet on a disk http://www.samizdat.com Sits with lots
of useful links: Savetz's Unofficial Internet Public Domain Index
Sorry I lost the pointer :-( but it is good Digital Libraries:
Electronic Journal and Text Archives
http://www.nlc-bnc.ca/ifla/II/etext.htm

Electronic Newspapers
=====================

The advent of electronic newspapers, on the net provides an excellent
opportunity to collect wordlists with minimal effort. The quality of
writing is usually good, they contain short articles on many
subjects, by many people, and so are an excellent source of words. A
disadvantage is that many are in an electronic, or unaccented version
of the language.

Listservers
===========

Many of the minor languages have BBS systems run from a University
computer.  Some are for learners, the spelling is awful. Some are
just chat, the spelling is generally bad.

Newsgroups
==========

The newsgroups soc.culture.* etc. are also a source of wordlists. The
writing and spelling are often of an appalling standard, and the text
is mostly unaccented.  Subjects are varied, but politics and religion
produce a very high proportion of postings, this produces highly
skewed lists, for instance in alt.culture.indonesia "allah" is in the
twenty most frequently used words. I do not believe that this is true
for normal indonesian text, or newspapers.

There are about 50,000 newsgroups on the net, but your server probably
only holds the 30,000 common ones. Many others are available which
typically are used by a single university. For some languages there
are whole hierarchies of newsgroups.  fr.*.* for French, de.*.* for
German. Even some of the Eastern European languages are developing
hierarchies cz.*.* contains some 20 newsgroups. Ask your sysop for a
list, you may find something interesting.

Paper Newspapers
================

All newspapers are now produced on computers, so if you can get
access to a newspapers files these will be fine. There will be short
pieces on lots of subjects by lots of journalists. The newspapers for
the masses write to a reading age of about 12, so have a restricted
range of words. A "quality" newspaper will be better, higher reading
age, and more subjects.

If possible take a few articles from many days issues. A single issue
may have a front page story, a leader article, and several pieces on
inside pages, all on the same subject, and therefore using the same
words.

Beware of the files produced by newspapers internally. A story goes
through many stages in the production cycle, so you can end up with
fifteen versions of the same thing!

There are an increasing number of sites containing links to Newspaper
Magazine and Media sites worldwide. Non have anything like a complete
list, so try several, and then use your favourite search engine.

     NEWSPAPERS, MAGAZINES AND PERIODICALS
     http://mulerider.saumag.edu/book/Communications.html
     MediaINFO Links - Search Page http://www.mediainfo.com/emedia/
     Magazines, Newspapers and Other Publications on the World Wide
     Web
     http://www.ca-probate.com/maglist.htm
     The Ultimate Collection of News Links http://pppp.net/links/news/
     Online Newspapers and Magazines of the World http://www.newsrack.com/

One can find text in many minority languages, correctly spelled and
accented and better sub edited. Even better, when using copy/paste
your browser will automatically change HTML accented characters into
your local character set. For languages which do not use latin1 you
may find that garbage is displayed, so you may have to reset your
browser language

IMHO these newspapers are the best source of text for the production
of wordlists at the time of writing 1/98. As the number of magazines
on line increases and reaches those for niche markets, it will be
possible create wordlists for different areas of interest.

Many paper newspapers publish many years issues on CDROMS.  These
cost money, but are an excellent source of well spelled words

'Zines
======

The internet has spawned a Genre called 'Zines, which are electronic
magazines published only or mainly on the net. They are normally run
by "not for profit" organizations, and each concentrates in a small,
sometimes very small interest group.  The quality of writing and
spelling are not as good as newspapers, because the sub editors do
not do it for a living

Academic papers
===============

There are a mass of academic papers on the net on every subject under
the sun. Use of these for wordlists is not recommended , as the
vocabularies used are specialized in the extreme.

Press offices
=============

Government Press offices etc. are now putting news on the net, and can
be used.  They however tend to concentrate on Political news, and
thus have a highly skewed, and small, selection of words, although
the quality of text is good.

Dictionaries
============

It may seem obvious, but dictionaries can be converted into wordlists.
Bilingual language1 to language2 dictionaries can with a bit of
very!! very!! nifty editing can be made into wordlists for either
language1 or language2. There is always a problem with dictionaries
that they contain words which are interesting to linguists, and not
words which are used in real text, or which people really speak.

Derived words: past and future tenses, plurals, gerunds, etc. etc. are
normally omitted. This is especially important in irregular verbs
etc., where the derived word is often completely different from the
root. Compound words are under represented, and in agglutinative
languages, such as German, are made up on the fly, and so cannot be
even listed. They are however fully accented and the words which they
contain are correctly spelled.

Online Dictionaries and Translators
http://rivendel.com/~ric/resources/dictionary.html

The DICT Development Group (www.dict.org) maintains the
English/American Dictionary Server Protocol, as described in RFC 2229
at http://www.dict.org. This includes wordlists of the most
frequently used ????? words, in American.

Things to edit out of text
==========================

When collecting text for wordlists, especially from internet, and with
number of occurrences, there are some things which are better edited
out.

     Headings and Headlines These are not representative of normal text.
     Repeated text On the net people often post things daily,
     weekly or as necessary. I have received the daily exchange rates
     for Slovenian currency, supplied courtesy of the Governor of the
     Bank of Slovenia.
     Salutations and signature names Gracias, hola, Hallo, hi etc. etc.
     should not be overepresented in word lists. Newsgroups always have
     frequent contributors, and mailing lists have owners. One does
     not want "Tony" or "Briony" overepresented.
     Signature blocks. These are idiosyncratic, and contain things
     representing the personality of the posters. The language is
     regularly not the same as the text above it.
     Quoted text. It is convention on internet to quote parts of a
     previous communication using > or : at the start of a line. This
     best omitted as it distorts  number of occurrence counts.

Accented or unaccented?
=======================

If at all possible collect words with correctly accented and modified
roman characters i.e. the slashed 'l' in polish, e acute in french.
The easiest words to find however are from internet, where it is
convention to use a computer version of the language which uses the
unaccented versions of characters.

There are more than 500 7 and 8 bit character sets available, so if
you can put in the readme file what, in the wordlist, indicated what
character!

Beware of the accented status of upper case characters, in some
languages, (French), by convention, lower case versions of a
character are accented, whereas upper case are often unaccented.

Non roman languages
===================

The vast majority of Russian, Bulgarian, Ukrainian, and Greek on
internet is transliterated into roman characters. Only a little is in
the correct characters. Several vowels in these languages
transliterate into the same roman character, so you cannot get simply
back to the correct character set from transliterated text. Greek
transliteration is a mess. Wordlists for non roman languages should be
in the original character set, those in transliterated versions are
of not much use.

How many words?
===============

For the simple non compounding languages, English, French, Italian
etc. about 50,000 to 100,000 words will cover almost everything used
in normal text. For the agglutinative languages where it is common to
add words to words to words, such as German, or Finnish, Several
hundred thousand are required, and worse the actual compound words
used keep changing. These are targets to aim for, wordlists of any
length are useful. A good rule of thumb is to continue collecting
until you need a lot of new text to add a few words to your wordlist.

Copyright
=========

IMHO as a non lawyer, the copyright of a piece of ordinary writing
rests in the artistic effort in writing it. This is totally lost when
you make a wordlist, so provided that you have the right to read it,
the copyright, of any wordlist produced, is yours and you can put it
in the public domain. Dictionaries may be different in that the
artistic effort is arguably in the selection of the words themselves.
There is a convention, that one dictionary is different from another
if every fifth word, is different, added, or deleted. Running two
equally sized wordlists together should satisfy this convention.

Readme file
===========

Each wordlist should have a readme file giving relevant information.
Please state who you are and your email address for queries, but the
following should be included:

     your name, snail mail address, Telephone number, and email address.
     the source of your list.
     any gross bias in the words found, political, religious, etc.
     The character set being used, with the representation or
     description of each of the accented or modified characters defined.
     Not everyone used TEX or your favourite code page! It is a real pain
     decoding the character set with a paper dictionary! There are
     more than 500, 7 and 8 bit character sets: ISO, PC, Mac, Gem,
     Word Perfect, ad infinitum. Not to mention 16, 32, 64 bit
     character sets, Unicode, ISO and so on.
     Say whether misspellings have been removed.
     Say whether loan words have been removed.
     They appear in all languages!!
     Say whether proper nouns are capitalised, capitalisation has been
     left as found, or all words have been lowercased. IMHO proper
     nouns are best capitalised if that is the convention in that language.

Number of occurrences
=====================

If one looks at the seminal work of the Brown corpus, available from
the Oxford Text Archive. One will find that the number of occurrences
of that word, in the text examined, is carefully tabulated. This is
very useful in many fields, and its addition to any wordlist is a
distinct advantage. If used for a spelling corrector, it is possible
to construct several dictionaries, one of common words held in main
memory, less common words held on disk and so on.


--
Dave Fawthrop <hyphen@hyphenologist.co.uk> <http://www.hyphenologist.co.uk>
Computer Hyphenation Ltd, Hyphen House, 8 Cooper Grove, Halifax HX3 7RF,
UK, Tel/Fax/Answer  +44 (0)1274 691092. **2000, 15th Anniversary Year**
Hyphenologist is sold as C source code and splits 50 languages.
Also on site: VDU Glasses, Wordlists FAQ, Celtic Spiral Font