🚀 go-pugleaf

	But I'm not sure how it got here.

In article <DASCHER.91Jun20154819@stanley.cis.brown.edu>, dascher@stanley.cis.brown.edu (David Ascher) writes:
> Hi all.
> 
> I'm about to start a very large newsfeed and use a good chunk of an
> rz57 as storage medium...  As usual, the problem that comes up is
> blocksize, since small blocksizes allow for an efficient storage of
> thousands of very small articles.  Problem is, it isn't "legal" to
> format an rz57 with a blocksize of less than 4k...  Which is a real
> problem.

	When the question was asked originally (internally) I
	wondered at the specific problem.  The first, and one
	I think the one you percieve is that it would nice to
	have block size of 2 KB.  Sorry.  4 KB and 8 KB are the
	only ones supported.

	The other problem, the answer to which you quote, is that
	on some disks many small files means you'll run out of
	inodes.  There is a newfs (mkfs) parameter that lets you
	control the number of inodes created.  The default value
	is 2048 bytes of disk/inode.  I've had exceptional success
	making it larger and getting fewer inodes, but not as much
	getting more.  That was a few versions ago and it may well
	have been broken.

	The work-around to this problem is:
> 
> Some people at DEC have told me that the only way around this is, and
> I quote:
> 
> "The solution to this problem is to lie about the geometry of the disk
> and tell it that there are more, but smaller, cylinders".

	More cylinders mean more groups of inodes and therefore
	more inodes.  If you have a habit of running out of inodes
	it's a reasonable solution.

	The real solution is the invention of the "Fat" file system.
	
> 
> Have any of you ever done this?  Any experts out there who can either
> tell me if my fears are unfounded, or where problems may crop up?

	Interesting question.  Are you're fears indeed unfounded?

	I looked my news disk (currently an RA82) and discovered
	161,581 files.  The average filesize is 2,978 bytes.  The
	distribution goes something like:

		Number	<= KB
		    31 	    0 (.04%)
		 38613      1 (24%)
		 74794      2 (46%)
		 25050      3 (16%)
		  9537      4 (6%)
		  4766      5 (and so on...)
		  2551      6
		  1210      7
 		   798      8

		  2095     16
 		   499     24
 		   305     32
 		   221     40
 		   309     48
 		   304     56
 		   426     64

		    71	>  64 KB (largest 1,204,224)

	So, a lot of the files will be less than 2 KB.  But a lot
	will be larger.  Now consider the fragment allocation
	scheme.  In one of the ULTRIX V4 releases we added (or at
	least documented) a flag to mkfs(8), to set the file system
	optimization flag.  The choices are to optimize for space
	or time.  In the case of space, the file system prefers to 
	allocate a file fragment with existing fragments, rather than 
	allocate a new fragment.  So, two 3 KB files will only use
	one 8 block on disk with 2 KB leftover.

	In the time case it allocates a new fragment.  This way if the
	file expands, it doesn't have to be copied to a new fragment
	with enough space.

	For news, I think the choice would be to optimize for space,
	which is also the default.  So, if you could choose a file
	system block size of 2 KB you constrain the file system to
	doing 2 KB I/O for any file, big or small.  Hundreds of small
	files will still packed as tightly together as possible.  If
	you use a larger block size, then reading back the larger
	files get the benefit of the larger block size.  I/O for the
	small files should only a block size that is "enough".

	I think your fears are unfounded.  If you can get enough inodes.
	
> 
> Thanks for any info.
> 
> -david ascher
>  Brown University Computing and Information Services
>  David_Ascher@Brown.EDU


-- 
Alan Rollow				alan@nabeth.cxn.dec.com