Re: Bug in EXCEL v. 2.2, and a flame

Article View: comp.sys.mac

Article #2199

#2199

From: Chris Cleeland
Date: Tue, 01 Aug 1989 00:16

23 lines
1546 bytes

In article <MYp9wBy00WB-8EBFJb@andrew.cmu.edu> jk3t+@andrew.cmu.edu (Jonathan King) writes:
>
>Interesting.  My calculator (a Sharp EL-512 II) gives virtually the same
>erroneous square root as Excel!  That is, the square root of .999975 is
>given as .999987499.  It looks like this problem is not unique to Excel;
>to make its way into the entirely silicon brain of my calculator this error
>must be the result of an ancient and honorable (although badly conditioned)
>numerics routine...
>

Funny, your Sharp and my HP must hail from a common genetic origin.  My
HP-28C give SQRT(.999975) as 0.999987499922; upon hitting the "x^2" key,
my trusty calc gives the original number -- .999975.  The 28C calculates
things to 12 digits of accuracy, and only displays 10 (presumably so that
all 10 are usable).  I hope that my calculator isn't wrong, but then
again, maybe I could blame some bad test grades on it ;->

Anybody else?
-- 
Thanks
Chris Cleeland, Tulane University
ADDRESS:  cleeland@rex.tulane.cs.edu
Disclaimer:  "I'm a student -- I can't afford to buy one!"

Technical Details

Message-ID: <959@rex.cs.tulane.edu>

Path: arc1-ca.pugleaf.net!utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!ames!rex!cleeland

References: <MYp9wBy00WB-8EBFJb@andrew.cmu.edu>

Show Full Headers

Complete Article Headers

Path: arc1-ca.pugleaf.net!utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!ames!rex!cleeland
From: cleeland@rex.cs.tulane.edu (Chris Cleeland)
Newsgroups: comp.sys.mac
Subject: Re: Bug in EXCEL v. 2.2, and a flame
Message-ID: <959@rex.cs.tulane.edu>
Date: 1 Aug 89 00:16:21 GMT
References: <MYp9wBy00WB-8EBFJb@andrew.cmu.edu>
Reply-To: cleeland@rex.UUCP (Chris Cleeland)
Distribution: usa
Organization: Computer Science Dept., Tulane Univ., New Orleans, LA
Lines: 23

Reply to Parent Message