🚀 go-pugleaf

[Was: Re: information about UNITREE?!?!  ]

FLAME RETARDANT:  Corrections and Additions are Welcome.




There are several packages out there which try to handle different
aspects of storage management.  These are the ones which I have heard of.
I don't know very much about most of them.  I would appreciate it if
those who do could fill in some of the blanks and post their additions,
comments, features, corrections, etc.

Hierarchical Storage Management - That is, providing a Unix filesystem
wherein "old" files are migrated offline or at least to media which have
relatively long access times (e.g. 3480 silos)

HSM Products:

** Cray Data Migration Facility (DMF) **
   - Cray only
   - 3480

** Unitree **
   - Multiple H/W S/W platforms - but which ones are "production"?
   - 3480, Metrum

** Convex Storage Manager (CSM) **
   - Convex only
   - 3480, 4mm DAT, HP Jukebox

** FileServ **
   - Multiple H/W S/W platforms
   - 3480, (+another high density medium?)



Archival and/or Backup - That is, taking a file or a directory tree, 
and storing it offline somewhere, but where it is accessible.  Two
key features can make a backup product "archival":  The ability of
users to archive and restore files without needing root, and the
ability of users (and root) to see what has been archived.  Otherwise,
archival is similar to backup.


Archival and Backup Products:

** Legato Networker **
   (platforms, media, features?)

** Epoch **
   (platforms, media, features?)


Backup (and Restore) Only, or archive manually:

dump/restore
tar
cpio  (everyone's favorite, of course ;-)  )







Addenda and Notes:

[1] There was an article in the early 1980's in IEEE Computer which
suggested that files be ordered for migration using (age*size).  This general
rule is good, but may need modification, particularly to accommodate 
filesystems with lots of files and directories.  Some users may need other
policies based strictly on age, or size, or some other policy.  Clearly a
system needs a flexible, programmable migration strategy to accommodate user
requirements.  

[2]  A system with "hierarchical" or "tree-structured" directories, (e.g. Unix),
needs additional sophistication to keep files grouped together, otherwise:
In my personal experience, you can, unfortunately, make repeated
media accesses in order to retrieve a bunch of (mixed size, mostly small)
files, all in the same directory.  This is very inconvenient to the user and
also extremely wasteful of a critical resource: media accesses.  
Therefore, it is critical to:
   A)  Keep small files in a directory tree together with the directory tree
       all the other small files in the directory tree, and:
   B)  Intelligently read ahead and cache, expecting that when one file is
       accessed, others might be (you can put such files in a short term
       cache and throw them away if not used soon)
   C)  Expect to keep a copy of every directory tree separate from the main
       files, and expect that a lot of such directory trees will be on
       mass storage.  In other words, expect that users will frequently use
       "find", and accommodate it, rather than fight it.
   D)  "Batch" together chunks of directory trees and files into manageable
       pieces in the 10-100 MB range.  When directories get modified, re-archive
       all the directories in the chunk together.  When files get modified,
       re-archive everything in the chunk together, files and directories.
   E)  Garbage-collect.  When media become X% (e.g. 50%) obsolete, mark all the
       chunks on the (cartridge etc.) as "to be rewritten.  Re-write in the
       background, off-peak.  Recycle the media.

[3]  An HSM product isn't very useful if you can't export filesystems via 
NFS (today) and other (e.g. DFS in the near future) network filesystems.  
Users want transparent network access to their filesystems from all machines.
Hence, the filesystem had better be able to support that kind of access,
which also tends to be very directory and inode intensive.  One of the beauties
of tree-structured (hierarchical) directories is their ability to organize
information in a way that users can find it (more) easily.  The proliferation 
of large numbers of small files and their associated directories is the price 
paid for this.  The goal is to accommodate normal usage patterns, including
"find", etc., within a product which performs well for thousands of users.

[4]  Think big.  Successful products may need to inhabit a world in which
millions of users are each storing millions of files on a common Internet.



-- 
  Hugh LaMaster, M/S 233-9,     UUCP:      ames!lamaster
  NASA Ames Research Center     Internet:  lamaster@ames.arc.nasa.gov
  Moffett Field, CA 94035-1000  Or:        lamaster@george.arc.nasa.gov 
  Phone:  415/604-1056                     #include <usenet/std_disclaimer.h>