🚀 go-pugleaf

Phil Norris <pn@stc.UUCP> writes:
> Are there any lexicographers out there?  I believe that's the correct term.
> I need to find a list of the most commonly used English words in WRITTEN
> TEXT (not oral useage).  The most common 8000 would do nicely, to coin a
> phrase.  Note no smiley here, I'm serious about the 8000, but I suspect
> that after the first few hundred most common, there's not much to choose
> between them.  Also in the same vein, I need the most common English word
> endings, preferably ranked in order of probability.

You can get this kind of information from the Brown corpus.
That is a random sampling of 2000 500-word samples of writing
published in 1963, grouped into categories including fiction,
non-fiction, etc., all from (I believe) generally available
U.S. publications. It's a bit over a million words, because
each sample is carried to the end of the current sentence
instead of cut at 500 words. (I might have it backwards - it
could be 500 2000-word samples.)

The Brown corpus came out of Brown University in Rhode Island.
When I was doing some linguistics research a few years back,
I found a copy of the tape at the U of Toronto computing centre,
along with a book that was published at the same time. The book
contains answers to some of the more common questions, such as
what the most common words are.

I also copied the tape (with dd) into an ASCII format, and wrote
a C program to find given words in context (a 5-line context).
There are some symbols (punctuation and paragraphing) which are
represented by asterisk codes and need to be programmed around.
If you do get hold of the Brown corpus, I could probably dig up
the source. (This was 1977-78, though.)

Check your local university library and/or computer centre.

Dave Sherman
The Law Society of Upper Canada
Osgoode Hall
Toronto, Canada  M5H 2N6
+1 416 947 3466
-- 
{ ihnp4!utzoo  pesnta  utcs  hcr  decvax!utcsri  } !lsuc!dave