🚀 go-pugleaf

Here are the consolidated answers to the original question from
Tom Donaldson (tomd@dfhake.pls.com):

     Does anyone know of any commercial libraries, soon-to-be-available
     software, shareware, freeware, or any-kinda-ware for tokenization of
     Japanese?  Are there any accepted algorithms for tokenizing Japanese?


Jeff
INSOFT-L List Manager

*========================================================================*
 Jeff Bowyer                         EMail:
 Computing Center		     jbowyer@cis.vutbr.cz
 Technical University of Brno
 Udolni 19, 602 00 BRNO
 Czech Republic
*========================================================================*


Sender: Jon Wtte (d88-jwa@nada.kth.se)

As usual, Apple system software shines - it has a tokenization
scheme built-into its script manager, and when the user installs and
uses a japanese script, all tokenizations are performed automagically
(such as FindWord())

If you're not developing for the Macintosh, maybe Apple will license it
to you. That'd be expensive, though.


Sender: Chuck Houpt (chuck@lec.com)
        Language Engineering Corp.

The Consortium for Lexical Research (CLR) has a bunch of Japanese
dictionaires, lexical analyzers and grammars.  Below are some examples from
their catalog.  They have an FTP site (clr.nmsu.edu) with many free
linguistic dictionaries, grammars, etc. for English, Japanese and European
languages.  Some of the material is only available to members of CLR (in
the "members-only" directory).


                      CONSORTIUM FOR LEXICAL RESEARCH


This is the CLR paragraph description catalog as of: Mon, Aug 2, 1993.

These entries are short paragraph length descriptions of the various materials
in the CLR archives.

For longer descriptions of the materials in the archives, the "catalog-short"
file contains pointers to the expanded information files.

Please feel free to contact us if you have any questions or comments related
to this catalog.

------------------------------------------------------------------------------
  CONSORTIUM FOR LEXICAL RESEARCH
------------------------------------------------------------------------------

  Computing Research Lab
  New Mexico State University
  Las Cruces, NM 88003

   phone: (505) 646-5466
     fax: (505) 646-6218
   email: lexical@nmsu.edu
  bitnet: lexical at nmsu

----------------------------------------------------------------
JUMAN-MCC

Ftp Directory:  members-only/lexica/JUMAN-MCC

Juman is a program which segments Japanese into words and tags
these words with parts of speech.  It was produced at Kyoto
University and then heavily modified by researchers at MCC.
The tables used for tagging are generated by a prolog program,
but the program which actually does the tokenizing and tagging
is written in C, so that users do not need to have a working
prolog implementation if they just want to use Juman.

----------------------------------------------------------------
JUMAN-1.0

Ftp Directory:     members-only/lexica/JUMAN-1.0/

Juman-1.0 is the newest version of the segmenter-tagger for Japanese.
It includes the JUMAN-MCC version.  In addition, it provides a 120K
dictionary.  Other minor changes have been made so that the software
is more convenient to use.

----------------------------------------------------------------

Japanese Morphological Dictionary

Ftp Directory:     pub/lexica/jmorphdict/

This is a Japanese morphological dictionary with a search program
included for accessing it.  The documentation is all in Japanese.

----------------------------------------------------------------
KN-PARSER

Ftp Directory:  members-only/tools/ling-analysis/KN-parser/

This is a Japanese parser which detects either a dependency or a case
structure.  In the case of the dependency structure, a set of
heuristic rules are used to detect the unique structure of a sentence.
In the case of a case structure, the analysis is performed on a sample
set of sentences found in the case-frame dictionary. The current
case-frame dictionary contains less than 1,000 verbs.  This parser
makes use of gmake, gcc, and juman.

The authors of KN-parser are Profs. Sadao Kurohashi and Makato Nagao,
Kyoto University.