🚀 go-pugleaf

Hi

I have a brute force method for extracting a keyword list from an Endnote
database.  It is long involved, very painful, horrible, and I'm sure that
no one will ever use it on a regular basis except maybe once every six
months as a maintenance check.  I don't want any flames for this method,
because it is the worst possible way to do this and I am perfectly well
aware of this.

So there.

At least we understand where I stand on this method.  The instructions make
reference to specific licensed software.  You are welcome to try this using
any software that you have.  I am very interested in responses to the
methodology.  Responses can be over the net or E-mailed to me directly,
FURR@BYRON.U.WASHINGTON.EDU.


OVERVIEW:
	Export your database to a text file
	Open the text file with Excel or Word (or both)
	Sort the resulting file, isolating all the occurrences of the keywords
	Edit the list to appear as you wish.

TIME to achieve a reasonable list:	1-3 hours.
TIME to achieve a pretty list:	hours or days.



WHAT DO YOU GET IN THE END?
	A very long list of ALL occurences of your keyword phrases.  This means
that if you used HIV 342 times in your database as a keyword then the list
will have 342 occurences of HIV.  It takes a time to remove these multiple
occurences if you dont want them.  If you are looking a neat one of
everything list, BEWARE, that will take time with this method!


Here is a DESCRIPTION OF THE DATABASE that I tried this on:
	Num. of Records:				2,475
	Size of EN database:			1,400K
	Size when exported:			  750K
	Average Num of keywords per record: 50 characters


Here are the TOOLS that I used:
	Macintosh IIsi with 5 megs of RAM, under system 7.01
	MS Word 5.0
	MS Excel 4.0
	EndNote Plus 1.2.1


THEORY:
	  When exporting a database, EN creates a textfile storing the
information.  The textfile is essentially made up of paragraphs.  A
paragraph for our purposes is comprised of characters followed by a
carriage return (control m, or end of paragraph marker or any other of many
names).
	  EN creates an exported textfile with each datafield in its own paragraph
(except for keyword phrases which each get their own paragraph).  Word is
capable of sorting paragraphs, and so is Excel in the sense that each
"paragraph" of an Excel file is placed in a single cell.
	  The reason we are interested in a SORTING function is that it is a way
to group the fields that we DON'T want, so we can delete them. That leaves
the fields (or paragraphs) that we DO want--each paragraph containing a
keyword phrase.


Things to REMEMBER:
	==This is SLOW!!
	==This takes a lot of memory.  Do everything you can to conserve. Save as
you go.  
	==Excel imports text files linearly.  That is, it fills the first column
up and then says it doesn't have room for more.  So Excel is limited to
16384 paragraphs of information.  But it's damn fast sorting.  So you can
work on parts of your database at a time, then merge the results.
	==Word is limited in its sorting routine, but very adept at changing the
data, which helps pare down the database for sending to excel.

	==EVERYTHING you are doing is stored in text files.  Save all your files
as text and open applications first before opening the files.  You do not
need translators or special export utilities other than what already is
built into EN, a good word processor and a probably a good spreadsheet.


Now, here is a DETAILED DECRIPTION of how I did it to mine:

1.  Close all applications.  Do everything you can to free up RAM memory. 
(Turn on virtual memory, etc.)

2.  Open Endnote. Open the desired database.

3.  Choose an export style.  I use ~REFER.

4.  Choose EXPORT.  Give a name to your export file.  This took about 10
minutes to export a 1.4 MB database.

5.  Quit  Endnote.  Find your export file and lock it, or back it up.  You
might need it again if you change horses.

6.  Open WORD.

7.  Open the exported text file.

8.  Count how many paragraphs there are.
	Fewer than 15000 or so?  you now have the option of going on to step 12.
	More than 10000?  go to step 9.

9.  While in WORD chose REPLACE.
	(because all the data fields start with %, followed by a single letter,
and then the data, we can reduce the number of fields by combining
them--removing the returns between each.  BUT we want to make the Keyword
field unique.  In other words we are going to diddle the data so there
aren't so many paragraphs.)

The field containing the keywords is different because it has carriage
returns between each keyword.  That's fine, and actually we want that.

SO, you have chosen replace.  What are you gonna do?  Four things, and
you'll do them IN ORDER!:  

PAY ATTENTION!  This is the biggest time consumed in this whole
procedure!!!
You must save your work as you go.  If you have large data, sometimes the
REPLACE window will throw you out for lack of memory and say SAVE.  Do that
and you can continue on the partially completed task.

a.  FIND WHAT (percent)(letter k)(space) REPLACE WITH (absolutely nothing)

b.  FIND WHAT (percent)(letter x) REPLACE WITH (end-of-paragraph
marker)(percent)(letter x)  THIS HERE STEP CAN BE SKIPPED IF HAVE VERY,VERY
FEW ABSTRACTS.

c.  FIND WHAT (percent)(letter o) REPLACE WITH (end-of-paragraph
marker)(percent)(letter o)  THIS HERE STEP CAN BE SKIPPED IF HAVE VERY,VERY
FEW NOTES.

d.  FIND WHAT (end-of-paragraph marker)(percent) REPLACE WITH (absolutely
nothing)  THIS STEP CANNOT BE SKIPPED!


Reading the above correctly is absolutely important!  An EOP marker can be
inserted by choosing from the little pull down menus to the left in the
REPLACE menu.



10.  Count the number of paragraphs and break your document into as many
separate documents as you need so that no part has more than 15000 or so
paragraphs.  YOU CANNOT GO ON TO THE EXCEL STEP UNTIL THIS IS DONE!


11.  You are still in WORD.  If you have a small document, try to highlight
the whole document and choose SORT from the TOOLS menu. If the program
throws you out for not enough memory, then the only thing to do is open the
document from excel (RE-READ #10!) Remember that you are trying to isolate
the keywords and delete the useless information.  ALL of your paragraphs
now consist of one of two things :either individual keyword phrases, or
large blocks of crap beginning with the percent character.  SORTing these
paragraphs will isolate the Percent paragraphs at the beginning for easy
deletion.

12.  Close WORD, saving any changes that you want saved.  Open EXCEL.  Open
your text document from within EXCEL.

13.  Highlight the first column which contains your data.  Chose SORT. 
When you have begun to isolate the useless fields delete them and save as
you go.  If you have come to this step without making some changes with
WORD realize that there will be a couple of things in this file:  cells
that are empty; cells that have single keyword phrases that are in perfect
working order; cells that have data that start with the percent character. 
(REMEMBER TO LOOK IN THE CELLS THAT start with %K, because those are valid
keywords that you want in your list.

14.  Quit EXCEL and open WORD, then open your data file that has been
greatly pared down.  Now in WORD you should be able to begin the major work
of cleaning up the list:  removing any stray notes or abstracts, taking
care of duplicate uses of keywords; look for bogus keywords that need to be
corrected in the original EN database.





Thank you for reading.  I hope that this helps other users.  

Remember, I am very satisfied.  EN is the best there is at what it does,
but it has room for improvement.  NILES needs to address KEYWORD master
listing, (and field master listing in general); exporting selected fields,
and so on.

Christopher Garry, "fur"