Re: Bug in EXCEL v. 2.2, and a flame

Article View: comp.sys.mac

Article #2308

#2308

From: Bill Jefferys
Date: Wed, 02 Aug 1989 19:22

13 lines
1084 bytes

In article <MYp9wBy00WB-8EBFJb@andrew.cmu.edu> jk3t+@andrew.cmu.edu (Jonathan King) writes:
#
#Interesting.  My calculator (a Sharp EL-512 II) gives virtually the same
#erroneous square root as Excel!  That is, the square root of .999975 is
#given as .999987499.  It looks like this problem is not unique to Excel;
#to make its way into the entirely silicon brain of my calculator this error
#must be the result of an ancient and honorable (although badly conditioned)
#numerics routine...

The error only shows up after the 10th significant digit. You gave only 9.
0.99998749992187 == 0.999987499, to the significance shown.

Bill Jefferys

Technical Details

Message-ID: <16496@ut-emx.UUCP>

Path: arc1-ca.pugleaf.net!utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!csd4.milw.wisc.edu!cs.utexas.edu!ut-emx!bill

References: <MYp9wBy00WB-8EBFJb@andrew.cmu.edu>

Show Full Headers

Complete Article Headers

Path: arc1-ca.pugleaf.net!utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!csd4.milw.wisc.edu!cs.utexas.edu!ut-emx!bill
From: bill@ut-emx.UUCP (Bill Jefferys)
Newsgroups: comp.sys.mac
Subject: Re: Bug in EXCEL v. 2.2, and a flame
Message-ID: <16496@ut-emx.UUCP>
Date: 2 Aug 89 19:22:35 GMT
References: <MYp9wBy00WB-8EBFJb@andrew.cmu.edu>
Reply-To: bill@emx.UUCP (Bill Jefferys)
Organization: UTexas Computation Center, Austin, Texas
Lines: 13

Reply to Parent Message