Thread View: comp.std.unix
5 messages
5 total messages
Started by "David J. Fiander"
Fri, 14 Aug 1992 20:22
Embedded data in shell scripts?
Author: "David J. Fiander"
Date: Fri, 14 Aug 1992 20:22
Date: Fri, 14 Aug 1992 20:22
85 lines
3121 bytes
3121 bytes
Submitted-by: davidf@mks.com ("David J. Fiander")
All references are to POSIX 1003.2 D11.2, unless stated
otherwise.
The standard says (lines 11645--11651)
When the shell is using standard input and it invokes a
command that also uses standard input, the shell shall
ensure that the standard input file pointer points
directly after the command it has read when the command
begins execution. It shall not read ahead in such a
manner that any characters intended to be read by the
invoked command are consumed by the shell (whether
interpreted by the shell or not) or that characters
that are not read by the invoked command are not seen
by the shell.
This seems to be attempting to ensure that a script like this
foo=`sed 1q`
hi there
echo $foo
echo Done
When fed to the shell on standard input will generate the output
hi there
Done
But the standard also says that (lines 5467--5471)
When a standard utility reads a seekable input file and
terminates without an error before it reaches
end-of-file, the utility shall ensure that the file
offset in the open file description is properly
positioned just past the last byte processed by the
utility. For files that are not seekable, the state of
the file offset in the open file description for that
file is unspecified.
This means that the above script is _only_ guaranteed to work
in the case of
sh <script
since sed is required to ensure that the file offset of the
seekable file script is properly positioned when it exits.
cat script | sh
does _not_ have to work, according to the standard. Yes, the
standard ensures that sed will read the right data, but the
shell may miss a large chunk out of the middle of the script
because sed is permitted to perform a buffered read, and not
reset the file offset pointer.
In fact, the only case in which both 'sh < script' and
'cat script | sh' will give the same answer on all systems is
that case in which the embedded data is all at the end of the
file, and that _all_ of that data will be consumed by the last
command executed by the shell. That is:
.
.
.
filter -args
data
data
Where filter will use all of the data.
What is most unfortunate is that in order to support this one
case, there will have to be a lot of code in the shell which,
when the script is be read from standard input and standard
input is not a seekable file, reads the script unbuffered, a
single byte at a time, in order to ensure that the last command
in the file get the data it is expecting.
Is it really worth it?
Volume-Number: Volume 28, Number 93
Re: Embedded data in shell scripts?
Author: Marc W. Mengel
Date: Fri, 14 Aug 1992 21:08
Date: Fri, 14 Aug 1992 21:08
30 lines
1130 bytes
1130 bytes
Submitted-by: mengel@dcdmwm.fnal.gov (Marc W. Mengel)
davidf@mks.com ("David J. Fiander") writes:
>This seems to be attempting to ensure that a script like this
> foo=`sed 1q`
> hi there
> echo $foo
> echo Done
>When fed to the shell on standard input will generate the output
> hi there
> Done
This only works well on tty-style devices, where reads return
on end-of line boundaries. It is also why the shell has
here-documents, so you can say
foo=`sed 1q <<EOF
hi there
EOF`
echo $foo
echo Done
to to the job right in a script, pipeline, etc.
-------
Marc Mengel
mengel@fnal.fnal.gov
Volume-Number: Volume 28, Number 98
Re: Embedded data in shell scripts?
Author: David J. Fiander
Date: Mon, 17 Aug 1992 21:38
Date: Mon, 17 Aug 1992 21:38
22 lines
1016 bytes
1016 bytes
Submitted-by: davidf@mks.com (David J. Fiander) According to mengel@dcdmwm.fnal.gov (Marc W. Mengel) >It is also why the shell has >here-documents, so you can say > > foo=`sed 1q <<EOF > hi there > EOF` > echo $foo > echo Done > >to to the job right in a script, pipeline, etc. I'm quite aware of that. The problem is that the standard seems to be attempting to allow my example in scripts and, in my opinion, failing badly. - David Volume-Number: Volume 28, Number 101
Re: Embedded data in shell scripts?
Author: John R. Levine
Date: Tue, 18 Aug 1992 21:20
Date: Tue, 18 Aug 1992 21:20
33 lines
2034 bytes
2034 bytes
Submitted-by: johnl@iecc.cambridge.ma.us (John R. Levine)
>The standard says (lines 11645--11651)
>
> When the shell is using standard input and it invokes a
> command that also uses standard input, the shell shall
> ensure that the standard input file pointer points
> directly after the command it has read when the command
> begins execution. ...
I don't see how you can have it any other way. If the shell is reading
directly or indirectly from a terminal I certainly wouldn't be too pleased
to have the shell eat arbitrary amounts of text after every command. And
I'd be equally unhappy if I saved the input stream into a file and
couldn't replay it.
The efficiency argument seems to me to be a red herring. When you pipe
large numbers of commands into the shell, if there aren't any loops in the
command list, the time will most likely be dominated by the time spent
running the commands. If there are loops, the looping commands are read
once and buffered inside the shell anyway. I agree that the distinctions
between seekable and non-seekable files are ugly, but it's too late to
legislate them out of existence.
Quite a while ago someone (Dennis Ritchie?) suggested that it would have
been a good idea to provide in Unix a version of read() that would read up
to but not past a terminator byte, independent of the data source. That
would have let programs read a line at a time with reasonable efficiency.
Regards,
John Levine, johnl@iecc.cambridge.ma.us, {spdcc|ima|world}!iecc!johnl
Volume-Number: Volume 29, Number 4
Re: Embedded data in shell scripts?
Author: David J. Fiander
Date: Wed, 19 Aug 1992 19:58
Date: Wed, 19 Aug 1992 19:58
41 lines
2535 bytes
2535 bytes
Submitted-by: davidf@mks.uucp (David J. Fiander) According to johnl@iecc.cambridge.ma.us (John R. Levine) (in <16rpiqINNon4@ftp.UU.NET>): >I don't see how you can have it any other way. If the shell is reading >directly or indirectly from a terminal I certainly wouldn't be too pleased >to have the shell eat arbitrary amounts of text after every command. And >I'd be equally unhappy if I saved the input stream into a file and >couldn't replay it. The problem is that 'cat script | sh' is not going to work, since the file offset is undefined after the first call to a standard utility. So your script will run fine until the first external command which reads stdin is invoked. >The efficiency argument seems to me to be a red herring. When you pipe >large numbers of commands into the shell, if there aren't any loops in the >command list, the time will most likely be dominated by the time spent >running the commands. If there are loops, the looping commands are read >once and buffered inside the shell anyway. I agree that the distinctions >between seekable and non-seekable files are ugly, but it's too late to >legislate them out of existence. I think that there will be an efficiency hit in the case we are discussing. My big complaint is that the text of the standard looks like somebody was attempting to ensure that their favorite cute hack would continue to work, but got it a little bit wrong. Also, none of the shells to which I have access right now demonstrate the "standard" behaviour, so what was the rationale behind this particular bit of text? >Quite a while ago someone (Dennis Ritchie?) suggested that it would have >been a good idea to provide in Unix a version of read() that would read up >to but not past a terminator byte, independent of the data source. That >would have let programs read a line at a time with reasonable efficiency. Yes, this would be nice. Too late now, 'though. - David Volume-Number: Volume 29, Number 6
Thread Navigation
This is a paginated view of messages in the thread with full content displayed inline.
Messages are displayed in chronological order, with the original post highlighted in green.
Use pagination controls to navigate through all messages in large threads.
Back to All Threads