🚀 go-pugleaf

This is a summary for the following

> We have a heavy loaded DEcsystem 5500, Ultrix 4.4 machine; a few hours ago
> we had a load greater than 15 and the network interface hung; the console
> displayed "can't get mbufs"; I know little of mbufs, except that dealing
> with them is kind of black magic; after five minutes or so the machine
> recovered and we could login again and so on; has anyone else experienced
> this behaviour ?

All agreed that the problem is rare and due to a shortage of physical
memory; one solution is to modify in

	sys/machine/mips/vmparam.h

the KMEMUMAP symbol to the maximum value added (the kernel will believe you
have more memory)

	#define KMEMUMAP (HIGH_WATER*256+KMEMSLOP)

others all rely on careful kernel tuning (have a look to O'Reilly System
Performance Tuning, b Mike Loukides)

I'm also including a DEC answer; anyway, we're going to add memory to our
system (the 'last' upgrade until a new machine arrive...); thanks to all
answered and bye from Italy,

	andrea

                        ************************

[ULTRIX] Recommendations When System Hangs With "can't get mbufs" Messages


COPYRIGHT (c) 1988, 1993 by Digital Equipment Corporation.
ALL RIGHTS RESERVED. No distribution except as provided under contract.

Copyright (c) Digital Equipment Corporation 1993. All rights reserved
OP/SYS: ULTRIX    V4.3

SOURCE:     Digital Customer Support Center

PROBLEM:

Your system is a DS5900 with two (2) ethernet interfaces.  Your system hangs
occasionally with "can't get mbufs" message.  Mbuf usage can be checked with
the following command:

        > netstat -m

                  25 mbufs (data) allocated
                 129 socket structures allocated
                 171 protocol control blocks allocated
                 485 routing table entries allocated
                   3 interface addresses allocated
                   8 socket names and addresses allocated
                   6 DECnet opts allocated
                   1 DECnet Access Right allocated
                  29 RPC structures allocated

If your system has crashed, check the mbuf usage with the following command:
NOTE:  If your system just hangs, force a crash.

        > "netstat -m vmunix.X vmcore.X "

Mbuf usage can also be checked either online or from a crash using
"vmstat -K."


RECOMMENDATIONS:

1.  This is generally an indication of exceeding a resource. Check memory
    and swap using the vmstat and pstat commands.

    Other possibilities: Try increasing ifqmaxlen from dbx.

        Example:
        ----------------------------------------------------------------------
        dbx -k /vmunix /dev/mem
                (dbx) p ifqmaxlen
                0x32
                (dbx) assign ifqmaxlen=250
                0xfa
        ----------------------------------------------------------------------
        If ifqmaxlen = 512, test using 1024 instead of 512.


2.  The problem may be relative to system tables.  To check this, issue the
    following command:

        > /etc/pstat -T
                237/1992       files
                579/1404       gnodes
                91/1056       processes
                37/ 102/ 172  active/reclaimable/total texts
                2/1349       00k swap

    Try increasing maxusers and rebuild the kernel.


3.  Many times this can be narrowed down to a network application or program
    which is causing the problem.  Check the number of errors and collision
    rate using:

        > netstat -i

    If you are seeing high errors and/or a high collision rate, you may have
    a problem with either your ethernet adapter or your network.  Another
    possibile indication of network problems is if the machine is a NFS
    server and is performing very poorly due to the load on the machine.

4.  In this particular case, the DS5900's culprit was DECmcc. DECmcc
    uses SQL.  The resource being exceeded was the file table (checked
    by using the "pstat -T" command.)

    We tried increasing the number of open files, configured in
    /sys/conf/mips/param.c.  The variable max_nofile is set to
    64 by default,so we increased this number and rebuilt the kernel.
    (We have tested this variable with a value as high as 4096.)

5.  Under ULTRIX V4.3 ,there is a patch possibly relative to this problem:

                /sys/MIPS/BINARY/sys_generic.o.

    The problem is the system allocates and never frees KM_NOFILE buffers.
    Note that vmstat -K reports these as FREE3 buffers. The problem appears
    to occur when a process has more than about 64 files open, then executes
    a select statement which is, in turn terminated by a signal.

Steps 4 and 5 are not specific to DECmcc, this can be applicable to any
database related product or any application that opens more than 64 files
per user.

RULE:  Use caution when increasing system tables, etc. Especially with a
       system that may not have enough physical memory.



From: Kursad Keskinege <kursad@bornova.ege.edu.tr>
Subject: Re: Ultrix 4.4 error: can't get mbufs

andrea,
We have faced with the same problem on a Dec Station 5000/25, under heavy 
load. Asked the same Q, and solved accordingly. I forward the answer 
that's useful:
-----------------
>From duncanl@cs.unc.eduTue Feb  6 16:49:18 1996
Date: Wed, 27 Sep 1995 15:30:36 -0400
From: Lynne Cohen Duncan <duncanl@cs.unc.edu>
To: kursad@baum01.ege.edu.tr
Subject: Re: cant get mbufs message

I had this problem with Ultrix 4.2a.  I don't know if the fix is the
same for 4.3.

Mail from Dec:
********************************************************************
"cant get mbufs" means you ran out of km_alloc map space, or you
ran short of physical memory (temporarily) and thus, dropped an
input packet.

You probably need to increase the km_alloc map on the system
experiencing the problem.   Adjust the value of KMEMUMAP in the 
sys/machine/vax/vmparam.h file.  [we modified sys/machine/mips/vmparam.h]

Background:

The message "cant get mbufs" is printed to the console by the kernel 
when a call to kmalloc has failed.  A kmalloc call can fail if there are 
not enough pte's [process table entries] available to map a given region 
of memory, or it can fail if there are pte's available but the physical 
memory is lacking at that particular instant.  A number of calls to the 
kmalloc kernel routing specify that the call is to wait for resources to 
become available should there be a temporary shortage.  The call in
question, which results in the "cant get mbufs" message does not specify 
that a wait should take place.  Thus, the kmalloc routine returns 
immediately with an error message, the kernel prints a message 
indicating the unavailability of resources, and the network packet is 
discarded.

This most likely occurs during a network "spike" whereby a host is 
flooded with hundreds of incoming packets during a brief instant, and it 
is unable to process all of the packets.

If this happens only randomly and without much frequency, you can ignore 
it.  If the machine hangs or locks up following the message, or if you 
see this message often, you should increases the value of KMEMUMAP.
********************************************************************


Me again:

KMEMUMAP is dependant on the value of PHYSMEM, which is set to 32 on
our servers.  The code in the vmparam.h file that specifies KMEMUMAP
is as follows:

#if (PHYSMEM/10 < LOW_WATER)
#define KMEMUMAP (LOW_WATER*256+KMEMSLOP)
#else
#if (PHYSMEM/10 > HIGH_WATER)
#define KMEMUMAP (HIGH_WATER*256+KMEMSLOP)
#else
#define KMEMUMAP (PHYSMEM/10*256+KMEMSLOP)
#endif
#endif

I changed it to be just the following:

#define KMEMUMAP (HIGH_WATER*256+KMEMSLOP)

This triples the actual value.

--------------
regards,


Kursad Keskinege, EgeNET
Services Coordinator
kursad@ege.edu.tr
http://www.ege.edu.tr/~kursad


-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Andrea Zuccollo is currently sitting at the Polytechnic of Milan
 Como site, located in via Castelnovo 7, 22100 Como, Italy.
 You can reach him by phone (+39 31 298 221), fax (+39 31 302 172) and
 e-mail (andrea@ing.unico.it)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~