Thread View: comp.sys.mac
5 messages
5 total messages
Started by Peter McCallum
Fri, 27 May 2005 08:03
calculator error?
Author: Peter McCallum
Date: Fri, 27 May 2005 08:03
Date: Fri, 27 May 2005 08:03
11 lines
906 bytes
906 bytes
When I use Calculator (v3.2.1) to do the following calculation: 9000 - 8387.7 - 557.3 I get the result: 54.999999999999 What gives? -- Peter McCallum Mackay Qld AUSTRALIA
Re: calculator error?
Author: Phil Wheeler
Date: Fri, 27 May 2005 10:25
Date: Fri, 27 May 2005 10:25
11 lines
1513 bytes
1513 bytes
Peter McCallum wrote: > When I use Calculator (v3.2.1) to do the following calculation: > 9000 - 8387.7 - 557.3 > > I get the result: > 54.999999999999 > Using v. 4.0 I get the same result. But changing the precision from 12 to 16 gives a more pleasing result. And so does changing the precision to 6. A bit odd.
Re: calculator error?
Author: Hans Aberg
Date: Fri, 27 May 2005 11:59
Date: Fri, 27 May 2005 11:59
24 lines
1636 bytes
1636 bytes
In article <1gx8fdy.xkc4ho1cgzvfwN%p5m8.REMOVETHIS@yahoo.com.au>, p5m8.REMOVETHIS@yahoo.com.au (Peter McCallum) wrote: > When I use Calculator (v3.2.1) to do the following calculation: > 9000 - 8387.7 - 557.3 > > I get the result: > 54.999999999999 > > What gives? Most computers nowadays converts all floating point numbers into binary form, and then performs operations on those. Thus, round-off errors might occur. You can get better precision in Terminal, by searching for calculator programs: man -k calculator One such program is "bc". You can learn about it by typing "man bc". Then type: echo "9000 - 8387.7 - 557.3" | bc 55.0 Here, the answer is merely a round-off of a higher precision result. -- Hans Aberg
Re: calculator error?
Author: Geoff Welsh
Date: Fri, 27 May 2005 16:59
Date: Fri, 27 May 2005 16:59
18 lines
1819 bytes
1819 bytes
Phil Wheeler wrote: > Peter McCallum wrote: > >> When I use Calculator (v3.2.1) to do the following calculation: >> 9000 - 8387.7 - 557.3 >> >> I get the result: >> 54.999999999999 >> > > Using v. 4.0 I get the same result. But changing the precision from 12 > to 16 gives a more pleasing result. And so does changing the precision > to 6. A bit odd. I noticed that about a year ago when I first purchased OSX. (Reminded me of using Fortan in College.) Anyway, changing the precision to 10 worked for me. Thanks for the tip! GW
Re: calculator error?
Author: Randy Howard
Date: Sat, 28 May 2005 01:16
Date: Sat, 28 May 2005 01:16
16 lines
1377 bytes
1377 bytes
In article <1gx8fdy.xkc4ho1cgzvfwN%p5m8.REMOVETHIS@yahoo.com.au>, p5m8.REMOVETHIS@yahoo.com.au says... > When I use Calculator (v3.2.1) to do the following calculation: > 9000 - 8387.7 - 557.3 > > I get the result: > 54.999999999999 > > What gives? http://docs.sun.com/source/806-3568/ncg_goldberg.html -- Randy Howard (2reply remove FOOBAR) "I don't really care about being right you know, I just care about success." --Steve Jobs
Thread Navigation
This is a paginated view of messages in the thread with full content displayed inline.
Messages are displayed in chronological order, with the original post highlighted in green.
Use pagination controls to navigate through all messages in large threads.
Back to All Threads