🚀 go-pugleaf

In <1992Aug29.210553.8744@rhein-main.de>, From: vhs@rhein-main.de
(Volker Herminghaus-Shirai) writes:

> I need to design a retrieval system for ~15TB of data ...
> 
> Any kind of information is greatly appreciated (hints, pointers, ideas, etc.)

my master's work at berkeley was on how to incorporate tertiary storage
into a database system cleanly.  i changed a bunch of internal interfaces,
defined a new abstraction like the file system switch, and added support
for a 327 gigabyte sony optical disk worm jukebox.

under this architecture, the database system manages the tertiary storage
device directly, rather than using an external file system interface to
it.  the advantage is that the dbms can make intelligent clustering,
caching, and prefetching decisions, and can consider optical device
performance characteristic (like very long pick times) when it optimizes
a query.

the jukebox storage manager uses a magnetic disk cache and extent-based
allocation to speed up accesses to data.

a description of the architecture is included in the postgres papers,
available by anon ftp from postgres.Berkeley.EDU (128.32.149.1) in
pub/postgres-papers.tar.Z.

this won't answer all your questions; in particular, you would have to
figure out clustering and caching policies to guarantee the performance
bounds you list.  it may help you out in identifying problems with these
devices and things you need to think about if you're designing your own
storage manager.
					mike olson
					project sequoia 2000
					uc berkeley
					mao@cs.berkeley.edu