🚀 go-pugleaf

In article <1993Jan12.104142.1311@wega.rz.uni-ulm.de>,
huober_r@hal.rz.uni-ulm.de (Reiner Huober) writes:
|> We use a 4D35 with biosym software on it. This software has a
|> help directory with over 3900 file, one or two blocks in size each.
|> If you copy or remove such a directory, then it takes several hours.
|> 
|> Is this specific to IRIX or have other systems the same behaviour
|> if there are many files in one directory.

  You are exceeding the disk buffer cache. It is usually not a
good idea to have thousands of files in one directory. The directory
gets so big, that it does not fit into the memory reserved by
the kernel. As a result, each access to any single inode in this
directory will directly go to disk, which is very slow. So if you
want to copy a directory with 3900 files, each file a few inodes,
it can take hours. All OS see this effect more or less drastic.
The limit is about 1000 files for a "normal" IRIX kernel. When
you exceed this limit, you will see the performance degradation
that you describe. If you have 'sar', the system activity reporter,
installed, you can try to monitor the system using 'sar -u' and
'sar -b'. If the former shows high percentages of %wio and %wfs,
this indicates a disk buffer cache problem. If 'sar -b' also
shows low utilisation of 'rcache' and 'wcache', you have another
indicator.

  To improve your situation, you can try to build a new kernel with
an increased disk buffer cache. To do so, log in as 'root' and
cd into '/usr/sysgen/master.d'. Edit file 'kernel'. Look for a
line containing '#define NBUF'. Under IRIX 4.0.x NBUF is set to
0 for autoconfiguration. Replace 0 by some large nuber like 1024
(I believe the maximum is 1500). Then rebuild the kernel using the
'autoconfig' command and reboot.

  Be aware that increasing NBUF increases the permament memory
usage of the kernel. If you are low on memory anyway, you will
hurt your overall system performance. You might want to play with
the NBUF parameter to get the best result ("acceptable" performance
at lowest possible NBUF value). As a final remark, don't expect the
resulting performance on your "monster" directory to be to good.
It will be better, but not impressive. The best way to overcome
your problem is to restructure the directory.

  Hope this helps.

Martin
-- 
+---------------------------------+----------------------------------+
|Martin Knoblauch                 | Silicon Graphics GmbH            |
|Application Support              | Am Hochacker 3 - Technopark      |
|Silicon Graphics Computer Systems| W-8011 Grasbrunn-Neukeferloh, FRG|
|                                 | Phone: (+int) 89 46108-179 or -0 |
|                                 | Fax:   (+int) 89 46108-222       |
+---------------------------------+----------------------------------+
|Network: <knobi@sgi.com>         | V-Mail: 8935   | M/S: IMU-315    |
+--------------------------------------------------------------------+

In <1iueqiINN6kr@fido.asd.sgi.com> knobi@knobi.munich.sgi.com (Martin Knoblauch) writes:
>In article <1993Jan12.104142.1311@wega.rz.uni-ulm.de>,
>huober_r@hal.rz.uni-ulm.de (Reiner Huober) writes:
>|> We use a 4D35 with biosym software on it. This software has a
>|> help directory with over 3900 file, one or two blocks in size each.
>|> If you copy or remove such a directory, then it takes several hours.
>|> 
>|> Is this specific to IRIX or have other systems the same behaviour
>|> if there are many files in one directory.

>  You are exceeding the disk buffer cache. It is usually not a
>good idea to have thousands of files in one directory. The directory
>gets so big, that it does not fit into the memory reserved by
>the kernel. As a result, each access to any single inode in this
>directory will directly go to disk, which is very slow. So if you
>want to copy a directory with 3900 files, each file a few inodes,
>it can take hours. All OS see this effect more or less drastic.
>The limit is about 1000 files for a "normal" IRIX kernel. When
>you exceed this limit, you will see the performance degradation
>that you describe. If you have 'sar', the system activity reporter,
>installed, you can try to monitor the system using 'sar -u' and
>'sar -b'. If the former shows high percentages of %wio and %wfs,
>this indicates a disk buffer cache problem. If 'sar -b' also
>shows low utilisation of 'rcache' and 'wcache', you have another
>indicator.

>  To improve your situation, you can try to build a new kernel with
>an increased disk buffer cache. To do so, log in as 'root' and
>cd into '/usr/sysgen/master.d'. Edit file 'kernel'. Look for a
>line containing '#define NBUF'. Under IRIX 4.0.x NBUF is set to
>0 for autoconfiguration. Replace 0 by some large nuber like 1024
>(I believe the maximum is 1500). Then rebuild the kernel using the
>'autoconfig' command and reboot.

Just as a note (because I've been burned by this one recently),
the maximum for NBUF under 4.0.X is not 1500.  There is no set
maximum.  In addition to increasing NBUF, you might also want to
consider increasing NHBUF as well.  Make sure that NHBUF, however,
is a power of two.  A recommendation to try is setting NBUF to
1024 and NHBUF to 256, reconfiguring your kernel, and testing.
Values to try beyond that are NBUF set to 2048 and NHBUF set to 512.
Give those a shot.  It should increase your directory performance.
If you have more questions, give the hotline a call for some more
recommendations.

--Matt

---------------------------------------------+-------------------------------
Matt D. Robinson (Member, Technical Staff)   |      mattr@awesome.csd.sgi.com
Silicon Graphics, Inc.                       |      mattr@leland.stanford.edu
--
---------------------------------------------+-------------------------------
Matt D. Robinson (Member, Technical Staff)   |      mattr@awesome.csd.sgi.com
Silicon Graphics, Inc.                       |      mattr@leland.stanford.edu