Re: Rounding error?!?!

Article View: comp.lang.postscript

Article #1895

Back to comp.lang.postscript Previous Next

#1895

From: Ciaran McHale
Date: Tue, 08 Sep 1992 17:29

23 lines
1470 bytes

In <1992Sep8.153127.27371@cetia.fr> alastair@bugle.cetia.fr
(Alastair Adamson) writes:
>|> I wrote the following trivial program to test my disbelief.
>|> It simply prints the difference between the "y" value used in
>|> a "moveto" and the "y" value returned by "currentpoint".
>|> I am amazed by the magnitude of the resulting value. Can this
>|> simply be put down to rounding error? Does anyone have a better
>|> explanation. The test was done on a LaserWriter II NTX.
>
>Sorry, I forgot to give the result of the test!
>The ouptut I got was 6.10352e-5 which I feel is a long
>way off zero.

6.10352e-5 is 1/16384, i.e, 1/(2**14). This sounds like a rounding error
to me. I wouldn't worry about it since it amounts to about 1/54 of a
hardware pixel at 300dpi.


Ciaran.
-- 
Ciaran McHale                                                             ----
Department of Computer Science, Trinity College, Dublin 2, Ireland.       \bi/
Telephone: +353-1-7021539 FAX: +353-1-772204 email: cjmchale@dsg.cs.tcd.ie \/

Technical Details

Message-ID: <1992Sep8.172913.19761@dsg.cs.tcd.ie>

Path: arc1-ca.pugleaf.net!sparky!uunet!mcsun!ieunet!tcdcs!maths.tcd.ie!longvalley!cjmchale

References: <1992Sep8.145018.27114@cetia.fr> <1992Sep8.153127.27371@cetia.fr>

Show Full Headers

Complete Article Headers

Newsgroups: comp.lang.postscript
Path: arc1-ca.pugleaf.net!sparky!uunet!mcsun!ieunet!tcdcs!maths.tcd.ie!longvalley!cjmchale
From: cjmchale@dsg.cs.tcd.ie (Ciaran McHale)
Subject: Re: Rounding error?!?!
Message-ID: <1992Sep8.172913.19761@dsg.cs.tcd.ie>
Organization: DSG, Dept. of Computer Science, Trinity College Dublin
References: <1992Sep8.145018.27114@cetia.fr> <1992Sep8.153127.27371@cetia.fr>
Date: Tue, 8 Sep 1992 17:29:13 GMT
Lines: 23

Reply to Parent Message