Thread View: comp.infosystems.wais
1 messages
1 total messages
Started by Maximilian Ott
Tue, 05 Jan 1993 03:47
adding "dynamic types" to waisindex
Author: Maximilian Ott
Date: Tue, 05 Jan 1993 03:47
Date: Tue, 05 Jan 1993 03:47
1161 lines
42210 bytes
42210 bytes
I use Wais to index a directory tree where I through all kind of
information into. This includes text, images, postscript, ... It
always annoyed me that waisindex seems to assume that all files have
the same type.
Not anymore. I wanted to start the New Year with a bit of hacking and
added a new data type called "magic". Similar to file(1) and magic(5)
the indexer now looks at the first few bytes of a file and tries to
determine its type. The heuristic is very simple and may fail
miserably, but it was quick to implement and it seems to work for my
files.
The different types are described in the file MAGIC_FILE (to be
defined in Makefile). Each line describes one type with
numberOfBytes cookie waisType
The cookie can be either a string or a hex (need to start with 0x). A
short sample:
4 From rmail
3 GIF gif
2 %! ps
2 0x4d4d tiff
The enclosed shar file includes five new files (magic.[hc] irtypes.[ch]
magic) and the diffs for irtfile.c, irbuild.c and Makefile.
Apply the diffs (I'm not sure if I did it right)and change MAGIC_FILE
to wherever you want to store that file. "make" should build a new
indexer and a new server (only because of a new inv.a). Now running
waisindex with "-t magic" will try to determine the type of every file
before it gets indexed.
WHAT DID I DO:
irbuild.c:
The major change to irbuild.c was to isolate the endless if-else chain
to parse the "-t" flag into a separate function. This is necessary as
the different separator functions may need to be set for every file.
I moved this function out of irbuild.c and into irtypes.c to include it
into inv.a. I also added a few more image types to it (PPM, JFIF, ..)
irtfiles.c:
"-t magic" will simply set the type variable to "CHECK_OUT". This will
be checked at the beginning of index_text_file. If true the type of
the current file will be determined and the approbate variables will be
set.
magic.c:
The public function is char* get_magic_type(char* filename) which will
return the typename. The first time around the magic file will be read
to build an array with an entry (cookie, mask, typename) for each
type. This is very simple; fixed array size, unsigned long better be
at least 32 bit long. If you want to check more than just the first
four bytes you have to do some work.
Enjoy,
max
Sat Jan 2 1993
p.s. Working for a company which "owns" whatever I think, even on a
holiday I found it necessary to put some legal stuff into all the
files. If that causes any problems drop me a line and I'll clear it
with our legal guys.
-----
Max Ott max@ccrl.nj.nec.com phone: (609)951-2469
NEC C&C Research Princeton, NJ, 08540 fax: (609)951-2499
#--------------------------------CUT HERE-------------------------------------
#! /bin/sh
#
# This is a shell archive. Save this into a file, edit it
# and delete all lines above this comment. Then give this
# file to sh by executing the command "sh file". The files
# will be extracted into the current directory owned by
# you with default permissions.
#
# The files contained herein are:
#
# -rw-r--r-- 1 max 10164 Jan 3 17:34 irbuild.c.diff
# -rw-r--r-- 1 max 635 Jan 3 17:34 irtfiles.c.diff
# -rw-r--r-- 1 max 741 Jan 3 17:35 Makefile.diff
# -rw-r--r-- 1 max 997 Jan 3 17:22 irtypes.h
# -rw-r--r-- 1 max 10506 Jan 3 17:22 irtypes.c
# -rw-r--r-- 1 max 5253 Jan 3 16:58 magic.c
# -rw-r--r-- 1 max 622 Jan 3 16:57 magic.h
# -rw-r--r-- 1 max 326 Jan 3 18:15 magic
# -rw-r--r-- 1 max 3471 Jan 4 22:41 README.magic
#
echo 'x - irbuild.c.diff'
if test -f irbuild.c.diff; then echo 'shar: not overwriting irbuild.c.diff'; else
sed 's/^X//' << '________This_Is_The_END________' > irbuild.c.diff
X6,10d5
X<
X< See the readme.magic for the disclaimer regarding the changes made
X< by NEC USA, Inc.
X<
X< max@ccrl.nj.nec.com
X105d99
X< #include "irtypes.h"
X175d168
X<
X352,365c345,598
X<
X< /* the magic type requires the index routine to check the type
X< of each file seperately. */
X< typename = next_argument;
X< if(0 == strcmp("magic", next_argument)) {
X< type = "CHECK_OUT";
X< waislog(WLOG_MEDIUM, WLOG_INDEX,
X< "Using magic to determine the type of each file");
X<
X< } else
X< set_file_characteristics(next_argument, &type,
X< &separator_function, &header_function,
X< &date_function, &finish_header_function,
X< &check_for_text_file);
X---
X> if(0 == strcmp("groliers", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = groliers_separator_function;
X> header_function = groliers_header_function;
X> finish_header_function = groliers_finish_header_function;
X> }
X> #ifdef NeXT
X> else if(0 == strcmp("objc", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = wobjc_separator_function;
X> header_function = wobjc_header_function;
X> finish_header_function = wobjc_finish_header_function;
X> }
X> #endif /* def NeXT */
X> else if(0 == strcmp("mail", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = mail_separator_function;
X> header_function = mail_header_function;
X> date_function = mail_date_function;
X> finish_header_function = mail_finish_header_function;
X> }
X> else if(0 == strcmp("mail_or_rmail", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = mail_or_rmail_separator;
X> header_function = mail_header_function;
X> date_function = mail_date_function;
X> finish_header_function = mail_finish_header_function;
X> }
X> else if(0 == strcmp("mail_digest", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = mail_digest_separator_function;
X> header_function = mail_header_function;
X> date_function = mail_date_function;
X> finish_header_function = mail_finish_header_function;
X> }
X> else if(0 == strcmp("mh_bboard", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = mh_bboard_separator_function;
X> header_function = mail_header_function;
X> date_function = mail_date_function;
X> finish_header_function = mail_finish_header_function;
X> }
X> else if(0 == strcmp("rmail", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = rmail_separator_function;
X> header_function = mail_header_function;
X> date_function = mail_date_function;
X> finish_header_function = mail_finish_header_function;
X> }
X> else if(0 == strcmp("netnews", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = NULL;
X> header_function = mail_header_function;
X> date_function = mail_date_function;
X> finish_header_function = mail_finish_header_function;
X> }
X> else if(0 == strcmp("rn", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = rn_separator_function;
X> header_function = mail_header_function;
X> date_function = mail_date_function;
X> finish_header_function = mail_finish_header_function;
X> }
X> else if(0 == strcmp("emacsinfo", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = emacs_info_separator_function;
X> header_function = emacs_info_header_function;
X> finish_header_function = emacs_info_finish_header_function;
X> }
X> else if(0 == strcmp("catalog", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = catalog_separator_function;
X> header_function = catalog_header_function;
X> finish_header_function = catalog_finish_header_function;
X> }
X> else if(0 == strcmp("bio", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = bio_separator_function;
X> header_function = bio_header_function;
X> finish_header_function = bio_finish_header_function;
X> }
X> else if(0 == strcmp("cmapp", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = cmapp_separator_function;
X> header_function = cmapp_header_function;
X> finish_header_function = cmapp_finish_header_function;
X> }
X> else if(0 == strcmp("ftp", next_argument)){
X> type = "TEXT-FTP";
X> typename = next_argument;
X> separator_function = first_line_separator_function;
X> header_function = first_line_header_function;
X> finish_header_function = first_line_finish_header_function;
X> }
X> else if(0 == strcmp("jargon", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = jargon_separator_function;
X> header_function = jargon_header_function;
X> finish_header_function = jargon_finish_header_function;
X> }
X> else if(0 == strcmp("server", next_argument)){
X> typename = next_argument;
X> type = "WSRC";
X> finish_header_function = filename_finish_header_function;
X> }
X> else if(0 == strcmp("text", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> check_for_text_file = true;
X> }
X> else if(0 == strcmp("filename", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> finish_header_function = filename_finish_header_function;
X> }
X> else if(0 == strcmp("irg", next_argument)){
X> typename = next_argument;
X> type = "TEXT";
X> separator_function = irg_separator_function;
X> header_function = irg_header_function;
X> finish_header_function = irg_finish_header_function;
X> }
X> /* dash-separated items , Intro to Algorithms buglist, etc */
X> else if(0 == strcmp("dash", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = dash_separator_function;
X> header_function = dash_header_function;
X> finish_header_function = dash_finish_header_function;
X> }
X> /* one_line-separated items */
X> else if(0 == strcmp("one_line", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = one_line_separator_function;
X> header_function = one_line_header_function;
X> finish_header_function = one_line_finish_header_function;
X> }
X> /* blank line-separated items (paragraphs) */
X> else if(0 == strcmp("para", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = para_separator_function;
X> header_function = para_header_function;
X> finish_header_function = para_finish_header_function;
X> }
X> /* seeker items */
X> else if(0 == strcmp("seeker", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = seeker_separator_function;
X> header_function = seeker_header_function;
X> finish_header_function = seeker_finish_header_function;
X> }
X> /* medline format */
X> else if(0 == strcmp("medline", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = medline_separator_function;
X> header_function = medline_header_function;
X> finish_header_function = medline_finish_header_function;
X> }
X> /* refer format */
X> else if(0 == strcmp("refer", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = refer_separator_function;
X> header_function = refer_header_function;
X> finish_header_function = refer_finish_header_function;
X> }
X> /* first_line format */
X> else if(0 == strcmp("first_line", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = first_line_separator_function;
X> header_function = first_line_header_function;
X> finish_header_function = first_line_finish_header_function;
X> }
X> /* rlin items */
X> else if(0 == strcmp("rlin", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = rlin_separator_function;
X> header_function = rlin_header_function;
X> finish_header_function = rlin_finish_header_function;
X> }
X> else if(0 == strcmp("dvi", next_argument)){
X> typename = next_argument;
X> type = "DVI";
X> finish_header_function = filename_finish_header_function;
X> }
X> else if(0 == strcmp("ps", next_argument)){
X> typename = next_argument;
X> type = "PS";
X> finish_header_function = filename_finish_header_function;
X> }
X> else if(0 == strcmp("pict", next_argument)){
X> typename = next_argument;
X> type = "PICT";
X> finish_header_function = filename_finish_header_function;
X> index_contents = false;
X> }
X> else if(0 == strcmp("gif", next_argument)){
X> typename = next_argument;
X> type = "GIF";
X> finish_header_function = filename_finish_header_function;
X> index_contents = false;
X> }
X> else if(0 == strcmp("tiff", next_argument)){
X> typename = next_argument;
X> type = "TIFF";
X> finish_header_function = filename_finish_header_function;
X> index_contents = false;
X> }
X> /* BibTeX items */
X> else if(0 == strcmp("bibtex", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = bibtex_separator_function;
X> header_function = bibtex_header_function;
X> finish_header_function = bibtex_finish_header_function;
X> }
X> /* ?:? seperated hypertext items */
X> else if(0 == strcmp("nhyp", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = nhyp_separator_function;
X> header_function = nhyp_header_function;
X> finish_header_function = nhyp_finish_header_function;
X> }
X> else if(0 == strcmp("ziff", next_argument)){
X> type = "TEXT";
X> typename = next_argument;
X> separator_function = ziff_separator_function;
X> header_function = ziff_header_function;
X> finish_header_function = ziff_finish_header_function;
X> }
X> else{
X> panic("Don't recognize the '%s' type", next_argument);
X> }
________This_Is_The_END________
if test `wc -c < irbuild.c.diff` -ne 10164; then
echo 'shar: irbuild.c.diff was damaged during transit (should have been 10164 bytes)'
fi
fi ; : end of overwriting check
echo 'x - irtfiles.c.diff'
if test -f irtfiles.c.diff; then echo 'shar: not overwriting irtfiles.c.diff'; else
sed 's/^X//' << '________This_Is_The_END________' > irtfiles.c.diff
X6,10d5
X<
X< See the readme.magic for the disclaimer regarding the changes made
X< by NEC USA, Inc.
X<
X< max@ccrl.nj.nec.com
X65,66d59
X< #include "irtypes.h"
X< #include "magic.h"
X82d74
X<
X684,695d675
X<
X< if (0 == strcmp(type, "CHECK_OUT")) {
X< char *typename;
X<
X< typename = get_magic_type(filename);
X< waislog(WLOG_LOW, WLOG_INDEX, "This is a <%s> type file", typename);
X< set_file_characteristics(typename, &type, &separator_function,
X< &header_function, &date_function,
X< &finish_header_function,
X< &check_for_text_file);
X< }
X<
________This_Is_The_END________
if test `wc -c < irtfiles.c.diff` -ne 635; then
echo 'shar: irtfiles.c.diff was damaged during transit (should have been 635 bytes)'
fi
fi ; : end of overwriting check
echo 'x - Makefile.diff'
if test -f Makefile.diff; then echo 'shar: not overwriting Makefile.diff'; else
sed 's/^X//' << '________This_Is_The_END________' > Makefile.diff
X52,53d51
X< MAGIC_FILE = /usr/local/pub/lib/wais/magic # cookie database
X<
X85,87c83
X< lock.c \
X< irtypes.c \
X< magic.c
X---
X> lock.c
X113c109
X< INDEX_SRC= irbuild.c
X---
X> INDEX_SRC= irbuild.c
X139,141c135
X< lock.o \
X< irtypes.o \
X< magic.o
X---
X> lock.o
X237,249d230
X< magic.o: magic.c
X< $(CC) $(CFLAGS) -c -DMAGIC_FILE="\"$(MAGIC_FILE)\"" magic.c
X<
X< max_shar:;
X< #rm -f irbuild.c.diff
X< #diff irbuild.c irbuild.c-orig > irbuild.c.diff
X< #rm -f irtfiles.c.diff
X< #diff irtfiles.c irtfiles.c-orig > irtfiles.c.diff
X< #rm -f Makefile.diff
X< #diff Makefile Makefile-orig > Makefile.diff
X< mkshar irbuild.c.diff irtfiles.c.diff Makefile.diff\
X< irtypes.h irtypes.c \
X< magic.c magic.h magic README.magic
________This_Is_The_END________
if test `wc -c < Makefile.diff` -ne 741; then
echo 'shar: Makefile.diff was damaged during transit (should have been 741 bytes)'
fi
fi ; : end of overwriting check
echo 'x - irtypes.h'
if test -f irtypes.h; then echo 'shar: not overwriting irtypes.h'; else
sed 's/^X//' << '________This_Is_The_END________' > irtypes.h
X/*
X * irtypes.h --setting the characteristics of indexed files
X * Copyright (C) 1993 Maximilian Ott, NEC USA, Inc.
X
X No guarantees or restrictions. See the readme.magic file for the
X full standard disclaimer.
X
X max@ccrl.nj.nec.com
X
X * Author : Maximilian Ott
X * Created On : Sun Jan 3 16:51:11 1993
X * Last Modified On: Sun Jan 3 17:22:56 1993
X * Status : Unknown, Use with caution!
X * Version : $Revision$
X ****************************************************************/
X
X#ifndef _irtypes_h_
X#define _irtypes_h_
X
Xextern void
Xset_file_characteristics _AP((char *typename,
X char **type,
X boolfunc **separator_function,
X voidfunc **header_function,
X longfunc **date_function,
X voidfunc **finish_header_function,
X boolean *check_for_text_file));
X
X#endif /* _irtypes_h_ */
________This_Is_The_END________
if test `wc -c < irtypes.h` -ne 997; then
echo 'shar: irtypes.h was damaged during transit (should have been 997 bytes)'
fi
fi ; : end of overwriting check
echo 'x - irtypes.c'
if test -f irtypes.c; then echo 'shar: not overwriting irtypes.c'; else
sed 's/^X//' << '________This_Is_The_END________' > irtypes.c
X/* WIDE AREA INFORMATION SERVER SOFTWARE:
X No guarantees or restrictions. See the readme file for the full standard
X disclaimer.
X
X Brewster@think.com
X
X See the readme.magic for the disclaimer regarding the changes made
X by NEC USA, Inc.
X
X*/
X
X/********************************
X * The following code was extracted from <irbuild.c>.
X *
X * If we use the above routines to dynamically determine the type of a
X * file we have to set the various seperator functions from within
X * <irtfile.c>. The latter is part of <inv.a>. In the static case
X * these functions were assigned at the beginning of running waisindex
X * in <irbuild.c>. For clean linking this has to be moved into
X * <inv.a>. As we have to add this file as well we better do it right
X * now.
X */
X
X/*
X * Also added new types for PPM files and zilla's image format
X */
X
X#include <string.h>
X#include "panic.h"
X#include "cutil.h"
X#include "irtfiles.h"
X#include "ircfiles.h"
X
Xextern boolean index_contents;
X
Xvoid
Xset_file_characteristics(typename,
X type,
X separator_function,
X header_function,
X date_function,
X finish_header_function,
X check_for_text_file)
X
X char *typename;
X char **type;
X boolfunc **separator_function;
X voidfunc **header_function;
X longfunc **date_function;
X voidfunc **finish_header_function;
X boolean *check_for_text_file;
X
X{
X /* default settings */
X *separator_function = NULL; /* initailize to nil */
X *header_function = NULL;
X *date_function = NULL;
X *finish_header_function = NULL;
X *type = "TEXT"; /* default to text */
X index_contents = true;
X
X if(0 == strcmp("groliers", typename)) {
X *type = "TEXT";
X *separator_function = groliers_separator_function;
X *header_function = groliers_header_function;
X *finish_header_function = groliers_finish_header_function;
X }
X#ifdef NeXT
X else if(0 == strcmp("objc", typename)) {
X *type = "TEXT";
X *separator_function = wobjc_separator_function;
X *header_function = wobjc_header_function;
X *finish_header_function = wobjc_finish_header_function;
X }
X#endif /* def NeXT */
X else if(0 == strcmp("mail", typename)) {
X *type = "TEXT";
X *separator_function = mail_separator_function;
X *header_function = mail_header_function;
X *date_function = mail_date_function;
X *finish_header_function = mail_finish_header_function;
X }
X else if(0 == strcmp("mail_or_rmail", typename)) {
X *type = "TEXT";
X *separator_function = mail_or_rmail_separator;
X *header_function = mail_header_function;
X *date_function = mail_date_function;
X *finish_header_function = mail_finish_header_function;
X }
X else if(0 == strcmp("mail_digest", typename)) {
X *type = "TEXT";
X *separator_function = mail_digest_separator_function;
X *header_function = mail_header_function;
X *date_function = mail_date_function;
X *finish_header_function = mail_finish_header_function;
X }
X else if(0 == strcmp("mh_bboard", typename)) {
X *type = "TEXT";
X *separator_function = mh_bboard_separator_function;
X *header_function = mail_header_function;
X *date_function = mail_date_function;
X *finish_header_function = mail_finish_header_function;
X }
X else if(0 == strcmp("rmail", typename)) {
X *type = "TEXT";
X *separator_function = rmail_separator_function;
X *header_function = mail_header_function;
X *date_function = mail_date_function;
X *finish_header_function = mail_finish_header_function;
X }
X else if(0 == strcmp("netnews", typename)) {
X *type = "TEXT";
X *separator_function = NULL;
X *header_function = mail_header_function;
X *date_function = mail_date_function;
X *finish_header_function = mail_finish_header_function;
X }
X else if(0 == strcmp("rn", typename)) {
X *type = "TEXT";
X *separator_function = rn_separator_function;
X *header_function = mail_header_function;
X *date_function = mail_date_function;
X *finish_header_function = mail_finish_header_function;
X }
X else if(0 == strcmp("emacsinfo", typename)) {
X *type = "TEXT";
X *separator_function = emacs_info_separator_function;
X *header_function = emacs_info_header_function;
X *finish_header_function = emacs_info_finish_header_function;
X }
X else if(0 == strcmp("catalog", typename)) {
X *type = "TEXT";
X *separator_function = catalog_separator_function;
X *header_function = catalog_header_function;
X *finish_header_function = catalog_finish_header_function;
X }
X else if(0 == strcmp("bio", typename)) {
X *type = "TEXT";
X *separator_function = bio_separator_function;
X *header_function = bio_header_function;
X *finish_header_function = bio_finish_header_function;
X }
X else if(0 == strcmp("cmapp", typename)) {
X *type = "TEXT";
X *separator_function = cmapp_separator_function;
X *header_function = cmapp_header_function;
X *finish_header_function = cmapp_finish_header_function;
X }
X else if(0 == strcmp("ftp", typename)) {
X *type = "TEXT-FTP";
X *separator_function = first_line_separator_function;
X *header_function = first_line_header_function;
X *finish_header_function = first_line_finish_header_function;
X }
X else if(0 == strcmp("jargon", typename)) {
X *type = "TEXT";
X *separator_function = jargon_separator_function;
X *header_function = jargon_header_function;
X *finish_header_function = jargon_finish_header_function;
X }
X else if(0 == strcmp("server", typename)) {
X *type = "WSRC";
X *finish_header_function = filename_finish_header_function;
X }
X else if(0 == strcmp("text", typename)) {
X *type = "TEXT";
X *check_for_text_file = true;
X }
X else if(0 == strcmp("filename", typename)) {
X *type = "TEXT";
X *finish_header_function = filename_finish_header_function;
X }
X else if(0 == strcmp("irg", typename)) {
X *type = "TEXT";
X *separator_function = irg_separator_function;
X *header_function = irg_header_function;
X *finish_header_function = irg_finish_header_function;
X }
X /* dash-separated items , Intro to Algorithms buglist, etc */
X else if(0 == strcmp("dash", typename)) {
X *type = "TEXT";
X *separator_function = dash_separator_function;
X *header_function = dash_header_function;
X *finish_header_function = dash_finish_header_function;
X }
X /* one_line-separated items */
X else if(0 == strcmp("one", typename)) {
X *type = "TEXT";
X *separator_function = one_line_separator_function;
X *header_function = one_line_header_function;
X *finish_header_function = one_line_finish_header_function;
X }
X /* blank line-separated items (paragraphs) */
X else if(0 == strcmp("para", typename)) {
X *type = "TEXT";
X *separator_function = para_separator_function;
X *header_function = para_header_function;
X *finish_header_function = para_finish_header_function;
X }
X /* seeker items */
X else if(0 == strcmp("seeker", typename)) {
X *type = "TEXT";
X *separator_function = seeker_separator_function;
X *header_function = seeker_header_function;
X *finish_header_function = seeker_finish_header_function;
X }
X /* medline format */
X else if(0 == strcmp("medline", typename)) {
X *type = "TEXT";
X *separator_function = medline_separator_function;
X *header_function = medline_header_function;
X *finish_header_function = medline_finish_header_function;
X }
X /* refer format */
X else if(0 == strcmp("refer", typename)) {
X *type = "TEXT";
X *separator_function = refer_separator_function;
X *header_function = refer_header_function;
X *finish_header_function = refer_finish_header_function;
X }
X /* first_line format */
X else if(0 == strcmp("first", typename)) {
X *type = "TEXT";
X *separator_function = first_line_separator_function;
X *header_function = first_line_header_function;
X *finish_header_function = first_line_finish_header_function;
X }
X /* rlin items */
X else if(0 == strcmp("rlin", typename)) {
X *type = "TEXT";
X *separator_function = rlin_separator_function;
X *header_function = rlin_header_function;
X *finish_header_function = rlin_finish_header_function;
X }
X else if(0 == strcmp("dvi", typename)) {
X *type = "DVI";
X *finish_header_function = filename_finish_header_function;
X }
X else if(0 == strcmp("ps", typename)) {
X *type = "PS";
X *finish_header_function = filename_finish_header_function;
X }
X else if(0 == strcmp("pict", typename)) {
X *type = "PICT";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X else if(0 == strcmp("gif", typename)) {
X *type = "GIF";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X else if(0 == strcmp("ppm", typename)) {
X *type = "PPM";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X else if(0 == strcmp("jfif", typename)) {
X *type = "JFIF";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X else if(0 == strcmp("sunraster", typename)) {
X *type = "SUNRASTER";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X else if(0 == strcmp("sgi", typename)) {
X *type = "SGI";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X else if(0 == strcmp("zilla", typename)) {
X *type = "ZILLA";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X else if(0 == strcmp("tiff", typename)) {
X *type = "TIFF";
X *finish_header_function = filename_finish_header_function;
X index_contents = false;
X }
X /* BibTeX items */
X else if(0 == strcmp("bibtex", typename)) {
X *type = "TEXT";
X *separator_function = bibtex_separator_function;
X *header_function = bibtex_header_function;
X *finish_header_function = bibtex_finish_header_function;
X }
X /* ?:? seperated hypertext items */
X else if(0 == strcmp("nhyp", typename)) {
X *type = "TEXT";
X *separator_function = nhyp_separator_function;
X *header_function = nhyp_header_function;
X *finish_header_function = nhyp_finish_header_function;
X }
X else if(0 == strcmp("ziff", typename)) {
X *type = "TEXT";
X *separator_function = ziff_separator_function;
X *header_function = ziff_header_function;
X *finish_header_function = ziff_finish_header_function;
X } else
X panic("Don't recognize the '%s' type", typename);
X} /* end of set_file_characteristics */
________This_Is_The_END________
if test `wc -c < irtypes.c` -ne 10506; then
echo 'shar: irtypes.c was damaged during transit (should have been 10506 bytes)'
fi
fi ; : end of overwriting check
echo 'x - magic.c'
if test -f magic.c; then echo 'shar: not overwriting magic.c'; else
sed 's/^X//' << '________This_Is_The_END________' > magic.c
X/*@ make -k
X * magic.c -- tries to determine the type of a file
X * Copyright (C) 1992 Maximilian Ott, NEC USA, Inc.
X
X No guarantees or restrictions. See the readme.magic file for the
X full standard disclaimer.
X
X max@ccrl.nj.nec.com
X
X * Author : Maximilian Ott
X * Created On : Sat Jan 2 13:26:41 1993
X * Last Modified On: Sun Jan 3 16:58:05 1993
X * Status : Unknown, Use with caution!
X * Version : $Revision$
X *+********************************************************************
X * Description:
X
X * char* get_magic_type(char* filename) will return the typename of
X * filename. The first time around the magic file will be read to
X * build an array with an entry (cookie, mask, typename) for each
X * type. This is very simple; fixed array size, unsigned long better
X * be at least 32 bit long. If you want to check more than just the
X * first four bytes you have to do some work.
X
X * The different types are described in the file MAGIC_FILE (to be
X * defined in Makefile). Each line describes one type with
X
X numberOfBytes cookie waisType
X
X * The cookie can be either a string or a hex (need to start with
X * 0x). A short sample:
X
X 4 From rmail
X 3 GIF gif
X 2 %! ps
X 2 0x4d4d tiff
X
X *
X * HISTORY:
X *
X * Known Bugs:
X *
X * Functions Defined:
X *
X * char* get_magic_type(char* filename)
X *
X **-******************************************************************/
X#include <stdio.h>
X#include <string.h>
X#include "cutil.h"
X#include "futil.h"
X#include "panic.h"
X#include "version.h"
X
X#ifndef LINT
Xstatic char rcsid[] =
X "$Id$";
X#endif
X
X/******* DEFINITIONS *************************************************/
X
X#ifndef MAGIC_FILE
X#define MAGIC_FILE "/usr/local/pub/lib/wais/magic"
X#endif MAGIC_FILE
X
X/* if you increase MAXCOOKIELENGTH you better check the code as this
X is used to create a mask and a long might not be enough to hold it.
X I agree, this is is pretty ugly. */
X#define MAXCOOKIELENGTH 4 /* max number of bytes for cookie */
X
X#define MAXMAGICENTRIES 40 /* max number of entries in the magic
X file */
X#define MAXBUFLEN 256 /* max length of a line in the magic
X file. */
X#define MAXTYPESTRING 30 /* max length of a type string. */
X
X
X/******* USEFUL MACROS ***********************************************/
X
X/******* EXTERNAL VARIABLES ******************************************/
X
X/******* LOCAL VARIABLES *********************************************/
X
Xstatic boolean magic_file_read = FALSE; /* set to TRUE after magic
X file has been read */
X
Xtypedef struct {
X long cookie;
X u_long mask;
X char type[MAXTYPESTRING];
X} Magic;
X
XMagic magic_list[MAXMAGICENTRIES]; /* make an entry for every line in the
X magic file. */
Xint magic_list_length = 0; /* the number of entries in the
X magic_list. */
X
X/******* EXTERNAL FUNCTIONS ******************************************/
X
X/******* LOCAL FUNCTIONS *********************************************/
X
Xstatic void
Xread_magic_file()
X
X{
X FILE* fp;
X char buf[MAXBUFLEN];
X int length;
X int i;
X char cookie[30];
X Magic* m;
X
X if ((fp = fopen(MAGIC_FILE, "r")) == NULL)
X panic("Can't open magic file <%s>", MAGIC_FILE);
X
X waislog(WLOG_LOW, WLOG_INDEX, "Reading magic file <%s>",
X MAGIC_FILE);
X while(fgets(buf, sizeof(buf), fp) != NULL) {
X if (buf[0] == '#') /* comment */
X continue;
X m = &magic_list[magic_list_length];
X if (3 != sscanf(buf, "%d%s%s", &length, cookie, m->type))
X panic("Wrong format in magic file <%s>", buf);
X /* convert cookie to hex */
X m->cookie = 0;
X if (0 == strncmp(cookie, "0x", 2)) {
X if (1 != sscanf(cookie, "%i", &m->cookie))
X panic("Can't convert cookie to hex <%s>", cookie);
X } else {
X for (i = 0; i < length; i++)
X m->cookie = m->cookie * 256 + cookie[i];
X }
X for (i = 0; i < MAXCOOKIELENGTH - length; i++)
X m->cookie *= 256;
X m->mask = 0xffffffff;
X for (i = 0; i < MAXCOOKIELENGTH - length; i++)
X m->mask = m->mask ^ (0xff << 8 * i);
X waislog(WLOG_LOW, WLOG_INDEX,
X "length: %d\tcookie: (%x) %s\tmask: %x\ttype: %s",
X length, m->cookie, cookie, m->mask, m->type);
X magic_list_length++;
X }
X fclose(fp);
X magic_file_read = TRUE;
X} /* end of read_magic_file */
X
X
Xchar*
Xget_magic_type(filename)
X
X char* filename;
X
X{
X char* type = "text"; /* default */
X FILE* fp;
X char buf[MAXCOOKIELENGTH];
X long cookie;
X Magic *m;
X int i;
X
X if (!magic_file_read)
X read_magic_file();
X
X if ((fp = fopen(filename, "r")) == NULL)
X panic("Can't open file <%s>", filename);
X if (NULL == fgets(buf, sizeof(buf), fp))
X panic("Can't read cookie from file <%s>", filename);
X fclose(fp);
X
X cookie = 0;
X for (i = 0; i < MAXCOOKIELENGTH; i++)
X cookie = cookie * 256 + buf[i];
X
X m = magic_list;
X for (i = 0; i < magic_list_length; i++, m++) {
X if (0 == ((cookie ^ m->cookie) & m->mask))
X return (m->type);
X }
X return(type);
X} /* end of get_magic_type */
________This_Is_The_END________
if test `wc -c < magic.c` -ne 5253; then
echo 'shar: magic.c was damaged during transit (should have been 5253 bytes)'
fi
fi ; : end of overwriting check
echo 'x - magic.h'
if test -f magic.h; then echo 'shar: not overwriting magic.h'; else
sed 's/^X//' << '________This_Is_The_END________' > magic.h
X/*
X * magic.h -- tries to determine the type of a file
X * Copyright (C) 1992 Maximilian Ott, NEC USA, Inc.
X
X No guarantees or restrictions. See the readme.magic file for the
X full standard disclaimer.
X
X max@ccrl.nj.nec.com
X
X * Author : Maximilian Ott
X * Created On : Sat Jan 2 13:28:09 1993
X * Last Modified On: Sun Jan 3 16:58:04 1993
X * Status : Unknown, Use with caution!
X * Version : $Revision$
X ****************************************************************/
X
X#ifndef _magic_h_
X#define _magic_h_
X
Xextern char*
Xget_magic_type _AP((char* filename));
X
X#endif /* _magic_h_ */
________This_Is_The_END________
if test `wc -c < magic.h` -ne 622; then
echo 'shar: magic.h was damaged during transit (should have been 622 bytes)'
fi
fi ; : end of overwriting check
echo 'x - magic'
if test -f magic; then echo 'shar: not overwriting magic'; else
sed 's/^X//' << '________This_Is_The_END________' > magic
X#
X# Similiar to file(1) and magic(5) we look at the first few bytes in
X# a file to determine its type.
X#
X#bytes cookie wais type
X#
X4 From rmail
X3 GIF gif
X2 %! ps
X2 0x4d4d tiff
X2 0xffd8 jfif
X4 0x59a66a95 sunraster
X2 0x01da sgi
X2 0x5069 zilla
X2 0x5369 zilla
X2 P1 ppm
X2 P2 ppm
X2 P3 ppm
X2 P4 ppm
X2 P5 ppm
X2 P6 ppm
________This_Is_The_END________
if test `wc -c < magic` -ne 326; then
echo 'shar: magic was damaged during transit (should have been 326 bytes)'
fi
fi ; : end of overwriting check
echo 'x - README.magic'
if test -f README.magic; then echo 'shar: not overwriting README.magic'; else
sed 's/^X//' << '________This_Is_The_END________' > README.magic
X
X
XWARRANTY DISCLAIMER
X
XThis software was modified by NEC USA, Inc. and is distributed free of
Xcharge. It is placed in the public domain and permission is granted
Xto anyone to use, duplicate, modify and redistribute it provided that
Xthis notice is attached.
X
XNEC USA, Inc. provides absolutely NO WARRANTY OF ANY KIND with respect
Xto this software. The entire risk as to the quality and performance
Xof this software is with the user. IN NO EVENT WILL THINKING MACHINES
XCORPORATION BE LIABLE TO ANYONE FOR ANY DAMAGES ARISING OUT THE USE OF
XTHIS SOFTWARE, INCLUDING, WITHOUT LIMITATION, DAMAGES RESULTING FROM
XLOST DATA OR LOST PROFITS, OR FOR ANY SPECIAL, INCIDENTAL OR
XCONSEQUENTIAL DAMAGES.
X----------------------------------------------------------------------
X
XI use Wais to index a directory tree where I through all kind of
Xinformation into. This includes text, images, postscript, ... It
Xalways annoyed me that waisindex seems to assume that all files have
Xthe same type.
X
XNot anymore. I wanted to start the New Year with a bit of hacking and
Xadded a new data type called "magic". Similar to file(1) and magic(5)
Xthe indexer now looks at the first few bytes of a file and tries to
Xdetermine its type. The heuristic is very simple and may fail
Xmiserably, but it was quick to implement and it seems to work for my
Xfiles.
X
XThe different types are described in the file MAGIC_FILE (to be
Xdefined in Makefile). Each line describes one type with
X
X numberOfBytes cookie waisType
X
XThe cookie can be either a string or a hex (need to start with 0x). A
Xshort sample:
X
X4 From rmail
X3 GIF gif
X2 %! ps
X2 0x4d4d tiff
X
XThe enclosed shar file includes five new files (magic.[hc] irtypes.[ch]
Xmagic) and the diffs for irtfile.c, irbuild.c and Makefile.
X
XApply the diffs (I'm not sure if I did it right)and change MAGIC_FILE
Xto wherever you want to store that file. "make" should build a new
Xindexer and a new server (only because of a new inv.a). Now running
Xwaisindex with "-t magic" will try to determine the type of every file
Xbefore it gets indexed.
X
XWHAT DID I DO:
X
Xirbuild.c:
X
XThe major change to irbuild.c was to isolate the endless if-else chain
Xto parse the "-t" flag into a separate function. This is necessary as
Xthe different separator functions may need to be set for every file.
XI moved this function out of irbuild.c and into irtypes.c to include it
Xinto inv.a. I also added a few more image types to it (PPM, JFIF, ..)
X
Xirtfiles.c:
X
X"-t magic" will simply set the type variable to "CHECK_OUT". This will
Xbe checked at the beginning of index_text_file. If true the type of
Xthe current file will be determined and the approbate variables will be
Xset.
X
Xmagic.c:
X
XThe public function is char* get_magic_type(char* filename) which will
Xreturn the typename. The first time around the magic file will be read
Xto build an array with an entry (cookie, mask, typename) for each
Xtype. This is very simple; fixed array size, unsigned long better be
Xat least 32 bit long. If you want to check more than just the first
Xfour bytes you have to do some work.
X
XEnjoy,
X
Xmax
X
XSat Jan 2 1993
X
Xp.s. Working for a company which "owns" whatever I think, even on a
Xholiday I found it necessary to put some legal stuff into all the
Xfiles. If that causes any problems drop me a line and I'll clear it
Xwith our legal guys.
X-----
XMax Ott max@ccrl.nj.nec.com phone: (609)951-2469
XNEC C&C Research Princeton, NJ, 08540 fax: (609)951-2499
X
X
________This_Is_The_END________
if test `wc -c < README.magic` -ne 3471; then
echo 'shar: README.magic was damaged during transit (should have been 3471 bytes)'
fi
fi ; : end of overwriting check
exit 0
--
Max Ott max@ccrl.nj.nec.com phone: (609)951-2469
NEC C&C Research Princeton, NJ, 08540 fax: (609)951-2499
Thread Navigation
This is a paginated view of messages in the thread with full content displayed inline.
Messages are displayed in chronological order, with the original post highlighted in green.
Use pagination controls to navigate through all messages in large threads.
Back to All Threads