🚀 go-pugleaf

SARA has recently contributed two pieces of software to the CUGDUS
community. Both are in the 'systems' area.

1	dma_qsort
2	Qman

dma_qsort is a small sort replacing program which increases the performance
of the dmhit program in DMF.

Qman 
  NQS only offers a possibility to move jobs into specific queues
dependend on user-specified disk requirements. It does not look at
available diskspace and/or interacts with DMF. Further the
user-specified disk requirement does not tell on which filesystem the
diskspace will be used.  SARA implemented a utility called
'Qman'. It uses NQS's feature to move jobs which require more than a
certain amount of diskspace into special queues.
These batchqueues are never started so jobs remain in there until Qman
moves them to the 'real' execution queues. Qman scans the stopped queues
for jobs. When a job is found in these queues, Qman examins the
available diskspace and decides whether or not the jobs can be handed
over to NQS (by moving it into the execution queue). This has the same
function as a complex disklimit would/should be implemented in NQS. If
insufficient diskspace is available Qman adjusts the DMF warning level
to force file migration. It lowers this level down upto a configurable
amount of diskspace. This prevents that the disk consuming jobs push all
'normally online' files to tape. If still more jobs need diskspace these
are postponed until running diskconsumers finish and diskspace becomes
available or the files created by those diskconsumers change from
'active batch job' file to 'normally online' file (which are candidate
for migration). In order to prevent migration of files which are created
by the disk consuming jobs a simple protection mechanism has been made
between Qman and the warning script of DMF.
Qman schedules diskconsumers only upto a configurable size. Whenever a
jobs requires more diskspace than this limit allows, the operator has to
decide when the job (a diskhog) must be run. When this moment arrives Qman will
stop scheduling new disk consumers, wait for the current disk consumers
to finish and then start the diskhog. As soon as the diskhog finishes Qman will
resume normal operation.

  All information Qman uses is derived from standard Unicos software
(NQS, DMF, etc) There is no knowledge about the jobs. A few rules of
thumb are used:
 . If a user specified a limit it is pretty accurate.
 . If a user did not specify a limit the job will use only some percentage
   of the default value for this limit.


Qman features two main characteristics:

- It limits the total amount of diskspace that can be used by batchjobs
- It changes the DMF warning level in such a way that unwanted filemigration
  is prevented.




Gerben Jansen,

SARA (Academic Computing Services),

Kruislaan 415,
1098 SJ Amsterdam,
The Netherlands.

e-mail: gerben@sara.nl
phone: +31 20 5923000