🚀 go-pugleaf

In article <1990Sep11.134238.20218@dg-rtp.dg.com> monroe@dg-rtp.dg.com (Mark A Monroe) writes:
: I want to rip a large file into pieces, naming new files according
: to an ID string in the large file.  For example, the large file contains
: records that look like this:
: 
: xxx-00001239	data	data	data
: description
:        .
:        .
: (variable length)
:        .
: 						<---blank line
: xxx-00001489	data	data	data
: description
:        .
:        .
: (variable length)
:        .
: 						<---blank line
: xxx-00001326	data	data	data
: 
: When I find a line in the large data file that starts
: with "xxx-0000", I want to open a file named "xxx-0000<number>",
: like "xxx-00001489", and write every line, including
: the current one, into it.  When I see another "xxx-0000",
: I want to close the file, open a new file named for the new id 
: string, and continue writing.  At the end of the large data
: file, close all files and exit.
: 
: Any suggestions?  

In standard shell+awk+sed it's a bit hard because you run out of file
descriptors.  You could do something like run sed over your file
to turn it into a giant script of here-is commands, but that'll be
real slow.

You could do something like this:

while read line; do
    case "$line" in
    xxx-0000*) set $line; exec >$1;;
    esac
    echo "$line"
done

But how well that works depends on the vagaries of your echo command,
such as what it does with lines starting with '-', or containing '\c'.
You don't really want to do this on a machine where echo isn't a builtin.

If you have Perl, your fastest solution will be to say something like

perl -pe 'open(STDOUT,">$&") if /^xxx-0000\d+/' filename

Change > to >> if the keys aren't unique in your input file.

Larry Wall
lwall@jpl-devvax.jpl.nasa.gov

In article <1990Sep11.134238.20218@dg-rtp.dg.com>, monroe@dg-rtp (Mark A Monroe) writes:
| I want to rip a large file into pieces, naming new files according
| to an ID string in the large file.  For example, the large file contains
| records that look like this:
| 
| xxx-00001239	data	data	data
| description
|        .
|        .
| (variable length)
|        .
| 						<---blank line
| xxx-00001489	data	data	data
| description
|        .
|        .
| (variable length)
|        .
| 						<---blank line
| xxx-00001326	data	data	data
| 
| When I find a line in the large data file that starts
| with "xxx-0000", I want to open a file named "xxx-0000<number>",
| like "xxx-00001489", and write every line, including
| the current one, into it.  When I see another "xxx-0000",
| I want to close the file, open a new file named for the new id 
| string, and continue writing.  At the end of the large data
| file, close all files and exit.
| 
| Any suggestions?  

You didn't say "and I don't want it in Perl", so I'm considering
this solution fair game...

perl -pe 'open(STDOUT,">$1") if /^(xxx-\d+)/;' bigdatafile

Pretty durn simple.  The right tool for the job.  Yeah, you could do
it with an awk script feeding into a /bin/sh (or with a smarter awk),
but this is too easy.

Just another Perl hacker,
-- 
/=Randal L. Schwartz, Stonehenge Consulting Services (503)777-0095 ==========\
| on contract to Intel's iWarp project, Beaverton, Oregon, USA, Sol III      |
| merlyn@iwarp.intel.com ...!any-MX-mailer-like-uunet!iwarp.intel.com!merlyn |
\=Cute Quote: "Welcome to Portland, Oregon, home of the California Raisins!"=/

In article <9466@jpl-devvax.JPL.NASA.GOV> lwall@jpl-devvax.JPL.NASA.GOV (Larry Wall) writes:
>     echo "$line"

C'mon, Larry. You know that should be echo -n "$line$n" where $n has
been initialized to a newline. Or echo -n "$line"; echo.

---Dan

>I want to rip a large file into pieces, naming new files according
>to an ID string in the large file.  For example, the large file contains
>records that look like this:

>xxx-00001239	data	data	data
>description
 
Seems like the "csplit" command may be able to help you.
You may have to rename the namely created files later using
the first line to get the number id that you want.

Parag      

In article <22842:Sep1121:10:3390@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes:
: In article <9466@jpl-devvax.JPL.NASA.GOV> lwall@jpl-devvax.JPL.NASA.GOV (Larry Wall) writes:
: >     echo "$line"
: 
: C'mon, Larry. You know that should be echo -n "$line$n" where $n has
: been initialized to a newline. Or echo -n "$line"; echo.

Don't teach your grandmother to suck eggs.  Let's see you come up with
a solution that also works on all the \c machines.

Apart from "Install SVR4".  :-)

Larry

>>>>> On 11 Sep 90 13:42:38 GMT, monroe@dg-rtp.dg.com (Mark A Monroe) said:

Mark> I want to rip a large file into pieces, naming new files according
Mark> to an ID string in the large file.  For example, the large file contains
Mark> records that look like this:

Mark> xxx-00001239	data	data	data
Mark> description
Mark>        .
Mark>        .
Mark> (variable length)
Mark>        .
Mark> 						<---blank line
Mark> xxx-00001489	data	data	data
Mark> description
Mark>        .
Mark>        .
Mark> (variable length)
Mark>        .
Mark> 						<---blank line
Mark> xxx-00001326	data	data	data

Mark> When I find a line in the large data file that starts
Mark> with "xxx-0000", I want to open a file named "xxx-0000<number>",
Mark> like "xxx-00001489", and write every line, including
Mark> the current one, into it.  When I see another "xxx-0000",
Mark> I want to close the file, open a new file named for the new id 
Mark> string, and continue writing.  At the end of the large data
Mark> file, close all files and exit.


Mark,
   Here's an awk solution.
------------------------split.awk-----------------------

BEGIN{pcFile="/dev/null";}
/xxx-[0123456789]+/{close(pcFile);pcFile = $1;}
{print $0 >> pcFile;}

--------------------------------------------------------

execute it by
awk -f split.awk datafile

Hope this helps
--
Timothy R. Ouellette
NCR Microelectronics                   Tim.Ouellette@FtCollins.ncr.com
Ft. Collins, CO.        	       uunet!ncrlnk!ncr-mpd!bach!timo
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"If all the world is a stage, I want to run the trap door" -- P. Beaty

In article <1990Sep11.134238.20218@dg-rtp.dg.com> monroe@dg-rtp.dg.com (Mark A Monroe) writes:
>I want to rip a large file into pieces, naming new files according
>to an ID string in the large file.  For example, the large file contains
>records that look like this:
>
>xxx-00001239	data	data	data
>description
>       .
>       .
>(variable length)
>       .
>						<---blank line
>xxx-00001489	data	data	data
>description
>       .
>       .
>(variable length)
>       .
>						<---blank line
>xxx-00001326	data	data	data
>
>When I find a line in the large data file that starts
>with "xxx-0000", I want to open a file named "xxx-0000<number>",
>like "xxx-00001489", and write every line, including
>the current one, into it.  When I see another "xxx-0000",
>I want to close the file, open a new file named for the new id 
>string, and continue writing.  At the end of the large data
>file, close all files and exit.
>
>Any suggestions?  

Use context split "csplit(1)" to break up the file efficiently. Then
use head/cut/mv to rename the pieces.

csplit -f aaa  /"^xxx-0000"/  {99}  <in-file-name>
rm aaa00
for FN in `ls aaa*`
do
	NFN=`head -1 $FN | cut -d' ' -f1 `
	mv $FN $NFN
done

Bob Rager

The right tool for the job is NOT perl but 'csplit'.

In article <1990Sep11.134238.20218@dg-rtp.dg.com> monroe@dg-rtp.dg.com (Mark A Monroe) writes:
>I want to rip a large file into pieces, naming new files according
>to an ID string in the large file.  For example, the large file contains
>records that look like this:

>xxx-00001239	data	data	data
>description
>       .
>       .
>(variable length)
>       .
>						<---blank line
>xxx-00001489	data	data	data

The perl suggestions are the best if you have perl, and it can be
done directly in awk if you have the new awk that can close files. 
I did this sort of thing years ago using:
 awk |sh
where the awk program generates a stream something like:
 cat > filename1 <<\!EOF
 data
 !EOF
 cat > filename2 <<\!EOF
 data
 !EOF

Les Mikesell
les@chinet.chi.il.us

In article <26116@boulder.Colorado.EDU> skwu@spot.Colorado.EDU.Colorado.EDU (WU SHI-KUEI) writes:
: The right tool for the job is NOT perl but 'csplit'.

"Those words fall too easily from your lips."  --Gandalf

Let us attempt to distinguish fact from dogma.

    1)  As far as I can tell, csplit is AT&T proprietary.  I certainly
	don't have it on all my machines, and don't know offhand where
	I'd find the source for it.  The person we were advising may
	well not have it on his machine.  You should at least say "If
	you have csplit..."

    2)	The man page for csplit (in the AT&T universe of a Pyramid, anyway)
	indicates that you can have a maximum of 99 output files.  The
	application in question could easily have more than that, judging
	by how it was specified.  A general tool should not have
	such limitations.

    3)	csplit won't name the files in the way specified--you'd have to
	follow it up with a loopful of mv commands, one process per file.
	And in the naive implementation, you'd have a sed or awk for each
	file to extract out the filename to hand to mv.

    4)	csplit can't recognize patterns across newlines (not that this
	job required that, but a general tool shouldn't have such
	limitations.)

    5)	csplit can get confused on lines longer than 255 chars.  It can't
	handle embedded nulls.  A general tool should not have such
	limitations.

    6)	Even if I did manage to find a freely available source for csplit,
	I'd have to worry about recompiling it on all my different
	architectures.  That would be okay (after all, I have to do that
	with Perl too), but I have to do it for 50 blue jillion other little
	"must have" tools too.  I'd much rather compile Perl once on
	each architecture, rewrite csplit in Perl, throw it into my
	/u/scripts directory that's mounted everywhere, and never worry about
	recompiling csplit again.

So it's not quite so simple as all that.  You can chop down a tree with
a hatchet, but sometimes you want an industrial strength Swiss Army Chainsaw.
And sometimes not.  There's more than one way to do it.

Larry

In article <9466@jpl-devvax.JPL.NASA.GOV> lwall@jpl-devvax.JPL.NASA.GOV (Larry Wall) writes:
>In article <1990Sep11.134238.20218@dg-rtp.dg.com> monroe@dg-rtp.dg.com (Mark A Monroe) writes:
>: I want to rip a large file into pieces, naming new files according
>: to an ID string in the large file.  For example, the large file contains
>: records that look like this:
>: 
>: Any suggestions?  
>
>In standard shell+awk+sed it's a bit hard because you run out of file
>descriptors.
In nawk you can close the old file before you open the new.
-- 
Bob Peirce, Pittsburgh, PA				  412-471-5320
...!uunet!pitt!investor!rbp			rbp@investor.pgh.pa.us