Article View: comp.sys.mac
Article #2709Re: Bug in EXCEL v. 2.2, and a flame
From: Graeme Hiebert
Date: Tue, 01 Aug 1989 19:38
Date: Tue, 01 Aug 1989 19:38
45 lines
2225 bytes
2225 bytes
In article <MYp9wBy00WB-8EBFJb@andrew.cmu.edu> jk3t+@andrew.cmu.edu (Jonathan King) writes:
>
> Interesting. My calculator (a Sharp EL-512 II) gives virtually the same
> erroneous square root as Excel! That is, the square root of .999975 is
> given as .999987499. It looks like this problem is not unique to Excel;
> to make its way into the entirely silicon brain of my calculator this error
> must be the result of an ancient and honorable (although badly conditioned)
> numerics routine...
>
> jking
Very interesting indeed. My Sun gives
sqrt(0.9999750000000000000000000) = 0.9999874999218740234222409
and
0.9999749992187 * 0.9999749992187 = 0.9999499990624
Now, let's see, if I do the multiplication by hand, I get
0.9999749992187
0.9999749992187
---------------
69998249945309
799979999374960
999974999218700
19999499984374000
899977499296830000
8999774992968300000
89997749929683000000
399989999687480000000
6999824994530900000000
89997749929683000000000
899977499296830000000000
8999774992968300000000000
89997749929683000000000000
--------------------------
0.99994999906243906561042969
^^^^^^^^^^^^^^^
Well look at that. My Sun multiplies the same way I do. Shame on us.
-g
--
"I didn't sleep well last night. This girl kept knocking on my hotel room
door. After awhile, I had to get up and let her out."
-Henny Youngman
Graeme Hiebert (hiebert@mdivax1.uucp, ...{uunet,ubc-cs}!van-bc!mdivax1!hiebert)
Technical Details
Message-ID:
<1989Aug1.193850.4999@mdivax1.uucp>Path:
arc1-ca.pugleaf.net!utzoo!utgpu!watmath!att!tut.cis.ohio-state.edu!cica!ctrsol!uakari.primate.wisc.edu!indri!caesar!blake!uw-beaver!ubc-cs!van-bc!mdivax1!hiebertReferences:
<MYp9wBy00WB-8EBFJb@andrew.cmu.edu>Show Full Headers
Complete Article Headers
Path: arc1-ca.pugleaf.net!utzoo!utgpu!watmath!att!tut.cis.ohio-state.edu!cica!ctrsol!uakari.primate.wisc.edu!indri!caesar!blake!uw-beaver!ubc-cs!van-bc!mdivax1!hiebert From: hiebert@mdivax1.uucp (Graeme Hiebert) Newsgroups: comp.sys.mac Subject: Re: Bug in EXCEL v. 2.2, and a flame Message-ID: <1989Aug1.193850.4999@mdivax1.uucp> Date: 1 Aug 89 19:38:50 GMT References: <MYp9wBy00WB-8EBFJb@andrew.cmu.edu> Reply-To: hiebert@mdivax1.uucp (Graeme Hiebert) Organization: Mobile Data International, Richmond, B.C., Canada Lines: 45