🚀 go-pugleaf

From: Owner-OS2News@teamos2.org

Subject: OS2News: The "HPFS Gives You More Space than FAT" FAQ

This article explains how and why HPFS gives more usable free space on
a typical hard drive than the FAT file system.  You may have heard
that HPFS wastes space unless you're on a huge partition.  Well, it's
not true.  Here's why:

On HPFS, the partition's control structures consume:
	 16 sectors for boot code
	  2 sectors for anchor blocks (superblock and spare block)
	  2 sectors for first code page
	101 sectors for hot-fixes
	 80 sectors for emergency directory b-tree expansion (20 DirBlocks)
	  4 sectors per every 8MB of disk space (0.025% of total space)

	Plus pre-allocated directory storage.  This storage is 20
	sectors (10KB) for every 1M of disk space, with a minimum of
	200 sectors (100KB) and a maximum of 16000 sectors (8MB).

	Directory entries in HPFS take up 36 bytes (for files whose
	names are 5 characters or less) plus extra space (in multiples
	of 4 bytes) for longer names.

	So, for 8.3 filenames (keeping things comparable to FAT), each
	directory entry will consume 44 bytes (36 plus 8 for the extra
	7 chars).  These directory entries will come out of the
	pre-allocated directory storage space, where possible.

On FAT, the control structures consume:
	         1 sector for boot code
	up to 1024 sectors for 2 copies of the FAT tables
	up to   32 sectors for the root directory
	slack to round everything to an even cylinder boundary

	FAT directory entries consume 32 bytes each.  Root directory
	entries consume no extra space, since their storage is
	pre-allocated.

In general, one can ignore the HPFS pre-allocated directory storage
and the FAT root directory space.  This is because:
- It's not much space - FAT's 32 sectors is only 16KB.  On HPFS, it's
  more space, but it's still only 10K for every MB of disk - 1% of the
  total space.  And for large drives (over 800MB), it stops growing.

- As files are allocated, this space is used.  For instance, 8MB of
  directory 8.3 HPFS directory entries is approximately 186000 files!
  This means you can create 186000 8.3 files without allocating any
  additional sectors for directory storage.

  If you want to count this storage in the free-space estimates, you
  should then count the amount of space 190000 FAT files would
  consume.  (5.7MB)  Even counting FAT's 16K of root directory in the
  mix, it's still a lot of extra storage.

For the general calculations, I'll ignore directory storage
(pre-allocated, root, or otherwise) in my calculations.  At the end of
the article is a re-calculation of my example cases, taking directory
storage into account.

So, if you have a partition size of p, your total amount of free space
on it will be approximately (assuming 512 byte sectors, of course):

	FAT: p - (1024 sectors = 512K)
	HPFS: (p * 0.99975) - (201 sectors = 100K)

This is what is often advertised.  And it becomes clear that HPFS
doesn't give you as much free space, since its control structures grow
in proportion with the partition size, whereas FAT's don't.

But, when you start adding files, this changes:

	FAT's cluster size is 2K for partitions under 128MB,
	                      4K for partitions over 128MB but under 256MB,
	                      8K for partitions over 256MB but under 512MB,
	                 and 16K for partitions over 512MB but under  1GB,
	                     etc...
	HPFS's cluster size is always 512 bytes.

On average, 1/2 of a cluster (slack space) is wasted for any file.
Additionally, HPFS consumes one sector (512 bytes) per file as an
Fnode.  So, if you have f files on a partition, the amount of wasted
space (from slack) is:

	HPFS: (256 bytes + 1 sector = 768 bytes) * f
	FAT: 1K * f (for partitions under 128MB)
	     2K * f (for partutions over 128MB but under 256MB)
	     etc...

Taking both together, you can compute how much space you will gain or
lose for a given partition to switch from HPFS to FAT (or vice versa):

In all cases,

	unusable space = control structures + slack

	free space = total space - unusable space - data.

Since data is what you're storing, we can discount it.  That will be
consumed equally, regardless of what the file system is.  We want to
minimize unusable space, since that space is waste.  We can compre
the efficiency of the file system easily, by comparing the wasted
space.

For example, I have two partitions on my hard drive.  One is 65MB, and
the other is 466MB.  The 65MB partition has about 1700 files and
directories on it.  The 466MB partition has about 7200 files and
directories on it.  (These are actual figures from the computer on
my desk, next to me right now.)

So, the amount of unusable space (in KBytes) for the 65MB partition is
about:

	 FAT: 512 + (1700 * 1) = 2212 (about 2.2MB)
	HPFS: (.00025 * 66560) + 100 + (1700 * .75) =
	      18 + 100 + 1275 = 1393 (about 1.4MB)

	In this case, HPFS would allow an extra 819K data to be stored
	on the drive.  (0.00025 * 66560 = 16.64, but we have to round
	it up to a 2K increment.)  This is about 1.2% of the total
	partition size.

On the 466MB drive, the wasted space is about:

	 FAT: 512 + (7200 * 4) = 29312 (about 28.6MB)
	HPFS: (.00025 * 477184) + 100 + (7200 * .75) =
	      120 + 100 + 5400 = 5620 (about 5.5MB)

	In this case, HPFS would allow 23.1MB more data to be stored on
	the drive.  This is about 5% of the total partition size.

As you can see, the amount of free space gained over FAT increases
rapidly as the partition size increases.  As an extreme, one could
calculate the wasted space on a 2GB partition with 25000 files on it.
(Not uncommon among file servers):

	 FAT: 512 + (25000 * 16) = 400512 (about 391.1MB)
	HPFS: (.00025 * 2097152) + 100 + (25000 * .75) =
	      526 + 100 + 18750 =  19106 (about 18.7MB)

	Here, HPFS allows an extra 372MB to be stored.  This is 18% of
	the total partition size!

-----------------

But, you say, "you've been ignoring the pre-allocated directory space
all this time".  OK.  While coming up with the equation for the
general case is a bit ugly, I'll recompute it for the three examples I
used above.  In all cases, assume that the filenames are exactly 8.3
characters (12 chars - don't forget to include the .) long.  In both
cases, I'll assume that directories are packed as tightly into
pre-allocated space as possible.  I'm ignoring slack in directories,
since directories are files, and the slack's average will be computed
in the average slack for the file system.

Of course, sub-optimal packing of directory names on the FAT drive
will end up wasting more directory space.  And use of longer HPFS
names will use up more of the pre-allocated dir space (but judging by
the size of it, I don't think additional directory storage would be
allocated anyway.)

So, for a 65MB partition with 1700 files:

FAT: FAT tables             =  512K
     Root DIR               =   16K (512 filenames)
     Extra 1188 dir entries =   37K (32 bytes per entry)
     Slack space            = 1700K (2K clusters)
     ------------------------------
     Total "waste"          = 2265K (about 2.2MB)

HPFS: Fixed structures    =  100K
      Variable structures =   18K (9 8MB bands - 36 sectors)
      Pre-allocated dirs  =  650K (10K per 1MB. 15127 filenames)
      F-nodes             =  850K (512 bytes per file)
      Slack space         =  425K (256 bytes per file)
      ---------------------------
      Total "waste"       = 2043K (about 2MB)

In this case, HPFS allows an extra 222KB of data to be stored.  This
is about 0.3% of the partition's total size.

-----

For the 466MB drive with 7200 files on it:

FAT: FAT tables             =   512K
     Root DIR               =    16K (512 filenames)
     Extra 6688 dir entries =   209K
     Slack space            = 28800K (8K clusters)
     -------------------------------
     Total "waste"          = 29537K (about 28.8MB)

HPFS: Fixed structures    =   100K
      Variable structures =   118K (59 8MB bands - 236 sectors)
      Pre-allocated dirs  =  4660K (108450 filenames)
      F-nodes             =  3600K
      Slack space         =  1800K
      ----------------------------
      Total "waste"       = 10278K (about 10MB)

In this case, HPFS allows an extra 19259K (about 18.8MB) of data to be
stored.  This is about 4% of the partition's total size.

-----

For the extreme case of a 2GB partition with 25000 files on it:

FAT: FAT tables              = 512K
     Root DIR                = 16K (512 filenames)
     Extra 24488 dir entries = 765K
     Slack space             = 400000K (32K clusters)
     ---------------------------------
     Total "waste"           = 401293 (about 391.9MB)

HPFS: Fixed structures    =   100K
      Variable structures =   512K (256 8MB bands - 1024 sectors)
      Pre-allocated dirs  =  8000K (max size - 186181 filenames)
      F-nodes             = 12500K
      Slack space         =  6250K
      ----------------------------
      Total "waste"       = 27362K (about 26.7MB)

In this case, HPFS allows an extra 373931K (about 365MB) of data to
be stored.  This is about 17.8% of the partition's total size.

---------------------------------------

In other words, even when you complicate the mathematics with the
directory structures, HPFS still gives you more free space than FAT
volumes.  On the small (65MB) partition, it's not much (only 222K) but
as the partition gets larger and the number of files increases, the
difference in storage becomes incredible.

---------------------+--------------------------------------------------------
David Charlap        | The contents of this message are not the opinions of
david@visix.com      | Visix Software, or of anyone besides myself.
Visix Software, Inc. +------------------------------------------------+-------
Member of Team-OS/2  | Quantum mechanics: The dream stuff is made of. |
---------------------+------------------------------------------------+