🚀 go-pugleaf

CSYSPCN@mvs.oac.ucla.edu (CSYSPCN) writes:

>What are you using for archiving, and are you happy with it?

What do you mean by archiving?

To *me* archiving means saving files in some system where I can be
confident that they will be looked after for as long as I have asked
them to be. Once I have 'archived' the file I should be able to delete
the file from my account. It also implies that I should be able to get
the file back with a minimum of fuss at anytime. Although there may be
a delay before the file appears. You need to be able to specify how long
the file is to be kept.

We are relatively new to unix. (We moved from an IBM VM system 18months
ago.) On the IBM we had a home grown archive system that worked very
well, by the time we moved off the IBM we had nearly 100,000 files
in the archive. The physical archive was stored on 150 reels of 1/2"
tape, two copies of every file stored on different tapes.

Since moving to Unix we have been activly looking for a system to
perform archiving under unix. Sofar we have examined several system
but all of them turned out to be backup system rather than archive
systems (for example 'Legato Networker'.) I'm not knocking these
system, if you want a backup system then there are some great
products out there. The crucial thing that all these systems lack that
make them unsuitable for archiving is that you can't give a command
'take this file and look after it for 5 years' then immediately delete
the file.

Anyway we have now given up looking and are starting to implement our own.
Don't hold your breath though! It will be quite a while before anything
appears. The problem is that we are very short of staff and I will end
up doing this as well as my other two full time jobs ;-(

Here are some comments about the system in the order that I think of them.

1/ It will be based on a client/server model. Server running on a UNIX
  system. Clients will be Unix, PC, Mac, VMS implemented in that order.
2/ Use TCP/IP a protocol based on ftp. We intend to  hack existing ftp
   clients on PCs and Macs as a first shot at implementing clients 
   for those platforms.
3/ the system will use a commercial indexed file system to maitain
   a database of information about files and users of the archive.
   (we will try to write this so as it make it easy to use some other
   database software -- it is likely that we will have a separate 
   daemon to look after the database which should localise the code
   nicely.)
4/ Security will be similar to unix password system with the possibility
   of having trusted hosts where user athentification is not required.
5/ full directory structures (from the users home directory for  Unix
   and VMS or from root for PCs and macs) will be saved.
6/ initial implementation will archive to a large disk area. Then
   we will implement the mechanisms to save files on tape. We have
   not finally settled what media we will use but are leaning
   towards dat because of its fast random access should speed retrieval.
7/ When fully operational (i.e. when we get the tape stuff implemented)
   the system will always maintain two copies of all files.
8/ We will be happy to distribute this software under a GNU type license.
9/ expected time frame -- early next year. (assuming the The good Lord 
   willing, creek don't rise,.... ;-)
10/ Afterthought, It will allow you to keep multiple versions of a file
    a la VMS. (you can specify how many generations to keep for a given
    file.)
 
Anybody out there got any comments suggestions?

Cheers, Russell.
-- 
Russell Fulton, Computer Center, University of Auckland, New Zealand.
<rj_fulton@aukuni.ac.nz>

CSYSPCN@mvs.oac.ucla.edu (CSYSPCN) writes:
> What are you using for archiving, and are you happy with it?

We sell an archiving tool, called CenDar (stands for: centralised
data archiving):

  CenDar is a reliable data archiving system for use in a heterogeneous
  network with various UNIX- and VMS-based computer systems, inter-
  connected by a local area network using the TCP/IP protocol suite.
    The aim of CenDar is to provide users with a reliable tool that
  enables them to archive sets of related files (grouped in so-called
  `projects') from various computers in the network (the clients, or
  `archive nodes') to an archive device attached to the server, or
  `CenDar node'.
    During the archive process, Cendar stores meta-data on the archived
  files in a database on the server.  Thus, users can easily retrieve
  files and generate reports on the contents of the database.
    CenDar uses a consistent, powerful, easy-to-use user interface.  All
  operations are controlled via this interface.


Russell Fulton (russell@ccu1.aukuni.ac.nz) writes:
> What do you mean by archiving?
> 
> To *me* archiving means saving files in some system where I can be
> confident that they will be looked after for as long as I have asked
> them to be. Once I have 'archived' the file I should be able to delete
> the file from my account. It also implies that I should be able to get
> the file back with a minimum of fuss at anytime. Although there may be
> a delay before the file appears. You need to be able to specify how long
> the file is to be kept.

This is what CenDar was designed to do.  Users can specify per `project'
how long the meta-data on the archived files should stay in the database
(the `expiration time').

> Here are some comments about the system in the order that I think of them.
> 
> 1/ It will be based on a client/server model. Server running on a UNIX
>   system. Clients will be Unix, PC, Mac, VMS implemented in that order.

The server runs under SunOS and HP-UX.  CenDar currently supports SunOS,
HP-UX and VMS as clients, with ports to ULTRIX and AIX planned.

> 2/ Use TCP/IP a protocol based on ftp. We intend to  hack existing ftp
>    clients on PCs and Macs as a first shot at implementing clients 
>    for those platforms.

CenDar uses TCP/IP, and currently uses the FTP protocol suite to
transfer the files from the clients to the server.

> 3/ the system will use a commercial indexed file system to maintain
>    a database of information about files and users of the archive.

CenDar support a wide range of commercial databases, amongst which are:
C-ISAM, Sybase, ORACLE, Ingres, Informix.

> 4/ Security will be similar to unix password system with the possibility
>    of having trusted hosts where user athentification is not required.

CenDar recognises three types of users:  administrators, project
managers and normal users.
  Administrators can archive and retrieve any file.  Project managers
can archive and retrieve only files normally accessible (via the
standard protection-based access mechanism of the operating system).
Normal users cannot archive, they can only retrieve files normally
accessible.

> 5/ full directory structures (from the users home directory for  Unix
>    and VMS or from root for PCs and macs) will be saved.

A project definition can specify any set of files to be archived (these
can be on different clients!).  If a specified file happens to be a
directory, the whole directory hierarchy is archived.

> 6/ initial implementation will archive to a large disk area.

CenDar uses an optical disk juke-box as its archive device.

> 7/ When fully operational (i.e. when we get the tape stuff implemented)
>    the system will always maintain two copies of all files.

CenDar archives only one copy of any file.  It is trivial to copy the
archive medium currently used by CenDar, and store the copy off-line.

> 8/ We will be happy to distribute this software under a GNU type license.

CenDar is NOT in the public domain.  It is a product of West Solutions.

> 9/ expected time frame -- early next year.

Available NOW.

> 10/ Afterthought, It will allow you to keep multiple versions of a file
>     a la VMS. (you can specify how many generations to keep for a given
>     file.)

CenDar will NEVER delete a file from the archive.  Users can retrieve
any version of the file that was archived.


As you can see, we already have the product you are looking for.  The
only practical problem will be the support of this product in your
area.
  Our sales area is the Benelux (Europe), with possible sales in
England, Germany, France, Italy and Spain.  We would gladly help you by
providing you with the code necessary to run the product, but support
should be done via E-mail or telephone.

The price of the product depends on the size of the juke-box connected
to the Cendar node.  Juke-boxes supported are the HP juke-boxes (10GB,
20GB, 60GB, 100GB).

If you want any further information, please do not hesitate to contact:

  Coen van den Heuvel
  Sales and Marketing director
  West Solutions
  E-mail: coen@west.nl


Regards,
--
-- Gertjan van Oosten, gertjan@west.nl

In article <1992Apr8.220057.3978@ccu1.aukuni.ac.nz> russell@ccu1.aukuni.ac.nz (Russell Fulton;ccc032u) writes:
>...
>Since moving to Unix we have been activly looking for a system to
>perform archiving under unix. Sofar we have examined several system
>but all of them turned out to be backup system rather than archive
>...
>   the system will always maintain two copies of all files.
>8/ We will be happy to distribute this software under a GNU type license.
>...
>Russell Fulton, Computer Center, University of Auckland, New Zealand.
><rj_fulton@aukuni.ac.nz>

I feel the same way and like the stated objectives enough to suggest
and offer actual financial support for such an effort.  Is there any
existing mechanism which has ever been demonstrated to result in 
reliable working software for non-proprietary use?  I'd personally
offer $1000 toward such a system, although I'd try to get my campus
to pay it if I could.  Would anyone care to contribute any other 
funds, perhaps to be paid upon successful trial & inspection?  Are 
there HUNGRY programmers ready to attempt such an application?

There are ways to proceed which limit the risk to all participants.

Dick

In article <3442@ucsfcca.ucsf.edu> dick@ccnext.UUCP (Dick Karpinski) writes:
>In article <1992Apr8.220057.3978@ccu1.aukuni.ac.nz> russell@ccu1.aukuni.ac.nz (Russell Fulton;ccc032u) writes:
>>Since moving to Unix we have been activly looking for a system to
>>perform archiving under unix. Sofar we have examined several system
>
>I feel the same way and like the stated objectives enough to suggest
>and offer actual financial support for such an effort.  Is there any


I am impressed with your willingness to support such a project, but you
should be aware that there are a couple of commercial products that do
archiving as described in the requirements list (not included in this
message), that may satisfy your needs.

To name a few...
	UniTree
	TOFS
	Epoch

A few public or almost public domain systems also exist:
	BUMP
	NAStore

Discalmer: I have nothing to do with any of these products.

---------------------------------------------------------------------------
  Sid Bytheway, (801) 581-3989    !  Internet: udpsid@snow.usi.utah.edu
  Senior Systems Programmer       !  Bitnet:   UDPSID at UTAHUSI
  Utah Supercomputing Institute   !  US Mail:  150 Park Bldg, SLC, UT 84112
---------------------------------------------------------------------------

P.S.  For those who may be interested...  I am currently in the process of
      writing a system based in 4.3BSD that will provide automatic movement
      of data from the online filesystems to offline storage for my Master's
      thesis.  It is not an "archiver" per se, but rather an hierarchical
      storage manager.  (If anyone could send me cold hard cash I'm sure it
      would help in my development efforts. :-) )

As it was mentioned in an earlier posting, let me add a couple of words on NAStore.

NAStore is a system to provide a Unix based, network connected file system with the
appearence of unlimited storage capacity.  The system was designed and developed at
NASA Ames Research Center for the Numerical Aerodynamic Simulation program.  The goal
was to provide seemingly unlimited file space via transparent archival (or migration)
of files to removable media (3480 tapes) in both robotic and manual handlers.  Supported
files sizes exceed the 2 gigabyte limit on most systems.  Archived data is restored
when accessed by the user with each byte being available as soon as it is restored rather
than having to wait for the whole file as is the case with other archival systems.

The NAStore system has been used here for 3 years and is under ongoing development.
It is based upon Amdahl's UTS and runs upon an Amdahl 5880.  We have 200 gigabytes of
on-line disk and 6 terabytes of robotic tape.

If your care for more information, let me suggest the following reading:
	1989 Winter Usenix Conference Proceedings, see the article on RASH
	the last IEEE Mass Storage Symposium proceedings

If you are still interested, contact Dave Tweten, e-mail tweten@nas.nasa.gov
-- 

Bob Henderson
hender@nas.nasa.gov
NASA Ames Research Center

"My goal in life?  To find the winning lotto ticket on the sidewalk!"