🚀 go-pugleaf

textmaker@hotmail.com (L Metcalfe) wrote:

>
>Hello all!
>
>I've searched everywhere on the net for this without success. If you
>could help me, I'd be hugely grateful. 
>
>I'm looking for a program that generates an index on a lot of text,
>not just searching for all individual words and small phrases (as Alta
>Vista does) but large strings of words making up paragraphs and entire
>letters. 
>
>It would generate a list ranked according to number of successful hits
>and size of the string, preferably not counting hits of small strings
>within the body of larger strings which have occurred a decent number
>of times (to disallow double counting).
>
>More intelligent features like similarity of strings (words in a
>different order, part of speech or having a good proportion of the
>words), and perhaps even ascertaining meaning would be nice, but not
>required. 
>
>
>Unix would be helpful (it can prolly be able to achieve this on one
>command line?) or an MS-DOS or Windows (3.1, 95 or NT) utility would
>be fantastic! 
>
>
>Thank you,
>
>textmaker@hotmail.com
The last time I needed something to do this kind of text parsing under
UNIX, I wound up writing some 600 lines of C code to parse a 64
character "free-form" data entry field.  

The more sophisticated Web search engines have these capabilities, but
there's a LOT of code involved to handle the "word A + word B in the
same reference and give points for being close" algorithm.

Someone has the source for this type search, but I doubt that it's
free ;-)

Visit my source code page at http://www.mindspring.com/~johnecarter
Visit my favorite school at http://www.mindspring.com/~addison

Get PR-Tracker -- tracks problem reports, defects, bugs
INFORMATION:  http://www.prtracker.com/info.html
DOWNLOAD:     http://www.prtracker.com/download.html

On Sun, 19 Oct 1997 01:28:48 GMT, L Metcalfe (textmaker@hotmail.com) wrote:
: 
: I'm looking for a program that generates an index on a lot of text,
: not just searching for all individual words and small phrases (as Alta
: Vista does) but large strings of words making up paragraphs and entire
: letters. 
: 
You want a full-text database that does ranked and proximity queries, by
the sound of it.  We've produced a GPLware full-text database that does the
former, and which is being modified to do the latter.  It may not be
exactly what you want, but have a look.

http://www.kbs.citri.edu.au/mg

Regards,

William Webber
-- 
William Webber        Multimedia Database Systems, RMIT, Melbourne, Australia
wew@yallara.cs.rmit.edu.au                  Food.  Shelter.  Source code.
  "'This tree is certainly good for nothing,' said Tzu Chi. 'This is why it
  has grown so large.  Ah-ha!  This is the sort of uselessness that sages
  live by.'" --- _The_Book_of_Chuang_Tzu_

Sounds to me that what you are trying to generate is beyond the normal
index. Looks like what is commonly called a concordance. Problem with
a program like that is more what to leave out than what to include.
How do you determine the small modifying words such as 'a', 'the',
'or', 'and', etc.. And how do you determine what is important? Or what
makes up a key phrase? Still it is not an outrageous requirement. I
did a quick websearch using Yahoo, and found a site that makes what
they call a "Text Database" called (of all things) Concordance. Might
be worth checking out. Not sure if its what you want but would be
worth a look.

http://concord.lax.primenet.com/

Hope this helps.

Myrrdin

On Sun, 19 Oct 1997 01:28:48 GMT, textmaker@hotmail.com (L Metcalfe)
wrote:

>
>Hello all!
>
>I've searched everywhere on the net for this without success. If you
>could help me, I'd be hugely grateful. 
>
>I'm looking for a program that generates an index on a lot of text,
>not just searching for all individual words and small phrases (as Alta
>Vista does) but large strings of words making up paragraphs and entire
>letters. 

.....................
 
>
>
>Unix would be helpful (it can prolly be able to achieve this on one
>command line?) or an MS-DOS or Windows (3.1, 95 or NT) utility would
>be fantastic! 
>
>
>Thank you,
>
>textmaker@hotmail.com

L Metcalfe wrote:
> 
> Hello all!
> 
> I've searched everywhere on the net for this without success. If you
> could help me, I'd be hugely grateful.
> 
> I'm looking for a program that generates an index on a lot of text,
> not just searching for all individual words and small phrases (as Alta
> Vista does) but large strings of words making up paragraphs and entire
> letters.
> 
> It would generate a list ranked according to number of successful hits
> and size of the string, preferably not counting hits of small strings
> within the body of larger strings which have occurred a decent number
> of times (to disallow double counting).
> 
> More intelligent features like similarity of strings (words in a
> different order, part of speech or having a good proportion of the
> words), and perhaps even ascertaining meaning would be nice, but not
> required.
> 
> Unix would be helpful (it can prolly be able to achieve this on one
> command line?) or an MS-DOS or Windows (3.1, 95 or NT) utility would
> be fantastic!

I have implemented a number of search engines/index generators and I
don't quite understand what you wish to have as your index terms. Would
you take a moment and create the index terms from your request? It would
appear long enought to have "interesting" terms, and such a list would
illustrate just what you are trying to accomplish. 

Don't forget, the search engine user must be able to recreate the
process or have the search engine do it for him from his input in order
to have matches.

   ---------------------------------------------------------------
The remainder of this text is to satisfy the requirement of my news
posting program that there be more response text than included text. I
personally find this to be an offensive requirement, but I have no
choice in the matter since I do not maintain the news reader code.


More intelligent features like similarity of strings (words in a
different order, part of speech or having a good proportion of the
words), and perhaps even ascertaining meaning would be nice, but not
required.

Unix would be helpful (it can prolly be able to achieve this on one
command line?) or an MS-DOS or Windows (3.1, 95 or NT) utility would
be fantastic!

   ---------------------------------------------------------------- 
-- 
Bob Stearns
University of Georgia
is@groucho.dev.uga.edu
(706)542-5110