Rounding error?!?!

Thread View: comp.lang.postscript

3 messages

[T] Back to Threads [V] Tree View [G] Group Articles

3 total messages Started by Alastair Adamson Tue, 08 Sep 1992 14:50

#1892

Author: Alastair Adamson
Date: Tue, 08 Sep 1992 14:50

20 lines
982 bytes

I wrote the following trivial program to test my disbelief.
It simply prints the difference between the "y" value used in
a "moveto" and the "y" value returned by "currentpoint".
I am amazed by the magnitude of the resulting value. Can this
simply be put down to rounding error? Does anyone have a better
explanation. The test was done on a LaserWriter II NTX.

%!
/y 700 def
/x 100 def
/Times-Roman findfont 20 scalefont setfont
x y moveto
currentpoint exch pop
y sub
(                  ) cvs show
showpage

Thanks in advance for any help you can give me,
	Alastair Adamson
	alastair@cetia.fr

Re: Rounding error?!?!

#1893

Author: Alastair Adamson
Date: Tue, 08 Sep 1992 15:31

14 lines
1071 bytes

In article <1992Sep8.145018.27114@cetia.fr>, alastair@bugle.cetia.fr (Alastair Adamson) writes:
|> I wrote the following trivial program to test my disbelief.
|> It simply prints the difference between the "y" value used in
|> a "moveto" and the "y" value returned by "currentpoint".
|> I am amazed by the magnitude of the resulting value. Can this
|> simply be put down to rounding error? Does anyone have a better
|> explanation. The test was done on a LaserWriter II NTX.

Sorry, I forgot to give the result of the test!
The ouptut I got was 6.10352e-5 which I feel is a long
way off zero.

	Alastair Adamson
	alastair@cetia.fr

Re: Rounding error?!?!

#1895

Author: Ciaran McHale
Date: Tue, 08 Sep 1992 17:29

23 lines
1470 bytes

In <1992Sep8.153127.27371@cetia.fr> alastair@bugle.cetia.fr
(Alastair Adamson) writes:
>|> I wrote the following trivial program to test my disbelief.
>|> It simply prints the difference between the "y" value used in
>|> a "moveto" and the "y" value returned by "currentpoint".
>|> I am amazed by the magnitude of the resulting value. Can this
>|> simply be put down to rounding error? Does anyone have a better
>|> explanation. The test was done on a LaserWriter II NTX.
>
>Sorry, I forgot to give the result of the test!
>The ouptut I got was 6.10352e-5 which I feel is a long
>way off zero.

6.10352e-5 is 1/16384, i.e, 1/(2**14). This sounds like a rounding error
to me. I wouldn't worry about it since it amounts to about 1/54 of a
hardware pixel at 300dpi.


Ciaran.
-- 
Ciaran McHale                                                             ----
Department of Computer Science, Trinity College, Dublin 2, Ireland.       \bi/
Telephone: +353-1-7021539 FAX: +353-1-772204 email: cjmchale@dsg.cs.tcd.ie \/

Thread Navigation

This is a paginated view of messages in the thread with full content displayed inline.

Messages are displayed in chronological order, with the original post highlighted in green.

Use pagination controls to navigate through all messages in large threads.

Back to All Threads