🚀 go-pugleaf

In article <4APR199218564977@stars.gsfc.nasa.gov>, thompson@stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) writes...
>Has anyone come to any conclusions about the FITS compression proposal by
>Archie Warnock et al.?  I'm interested in the subject of data compression in

Well... I have <grin>.

>including ASCII and binary tables.  However, as I see it, this scheme also has
>the following consequences:
>	* Only lossless compression algorithms can be used, so that the
>	  compressed header can be reconstructed correctly.

Yes - this is intentional.  Remember that FITS is an interchange format, 
and as such, should represent the data stream accurately.  You want to 
do lossy compression?  Do it before putting the data in FITS...

That's a partially facetious answer, and I'd welcome solutions which (a) 
retain the flexibility and (b) add the ability to use lossy compression. 
Part of the reason for releasing the proposal publicly was to generate 
such discussion _before_ the proposal became carved in stone (or at 
least carved in A&A Supp).

>	* Information about the data in the compressed header should also be
>	  reproduced in the compression extension header, so that the user can
>	  understand what the file contains.  Since the same data is included
>	  twice, some loss of compression efficiency is engendered.

This redundancy is not mandatory, so if you're really concerned with
efficiency, you can omit the "logical" keywords in the extension header. 
But see below the next point...

>	* Compression schemes that work well on the data, may be inefficient in
>	  reducing the extension header.

Yes, but we're only talking (typically) about a few thousand bytes of
header.  If that's a substantial fraction of the data volume, the
compression may be inappropropriate. 

>	* Could the Warnock et al. proposal be changed to *not* compress the
>	  header by making the optional keywords LBITPIX, etc. mandatory
>	  instead, or would this make the proposal less flexible?

Of course it _could_ be changed.  That's why we presented it at the AAS
meeting in Albuquerque.  An early iteration of the proposal did just such
a thing, but we realized that, since there's no way to "nest"
extensions, this would effectively exclude compressing any extensions -
and tabular data tends to be highly compressible.  You'd need to come up 
with some way to add keywords to document the presence of a compressed 
extension, and now you're talking about a (possibly) long and 
complicated set.

There has been talk of some heirarchical schemes for extensions, and
that may resolve some of these problems, but my sense is that those
proposals aren't even as mature as the compression proposal - and it's
still an infant, too. 

More discussion, more ideas...

----------------------------------------------------------------------------
-- Archie Warnock                     Internet:  warnock@nssdc.gsfc.nasa.gov
-- Hughes STX                         SPAN:      NSSDC::WARNOCK
-- NASA/GSFC                          "Hail To The Redskins!"