🚀 go-pugleaf

In article <12805@mentor.cc.purdue.edu> hrubin@pop.stat.purdue.edu
(Herman Rubin) writes:

... The IBM hex FP is not that much worse than any other FP with 32-bit words.

The average precision of SP hex normalization, something like 22.5
bits (compared to the 24 bits of VAX or IEEE), does not determine the
overall precision of complicated numerical operations like FFTs;
instead the 21_bit worst case causes random truncation errors in
intermediate sums, and thereby eventually corrupts most of the
answers. The loss of precision is easily detectable when SP results
from 370-architecture-CPUs are compared with SP results computed on
VAXen and IEEE-FP machines. For signal processing applications where
the signals and results have high dynamic range (e.g., more than
10,000:1), hex-normalized SP FP must be viewed with suspicion.

--

Donald C. Wells             Associate Scientist        dwells@nrao.edu
National Radio Astronomy Observatory                   +1-804-296-0277
Edgemont Road                                     Fax= +1-804-296-0278
Charlottesville, Virginia 22903-2475 USA            78:31.1W, 38:02.2N