🚀 go-pugleaf

In article <1993Jan17.001105.7415@austral.chi.il.us> rrezaian@austral.chi.il.us (Russell Rezaian) writes:
>In article <MWS3SEHMH@linac.fnal.gov> matt@severian.chi.il.us (Matt Crawford) writes:
>>Gregory Gulik:
>>> I just got off the phone with Russell.  He says he will try
>>> to make the trip down to Clout tonight (from Naperville) to fix it.
>>
>>I guess he didn't, eh?  Youse guys need an on-campus cloutster again,
>>like yez useta have.
>
>It was snowing rather badly at the time.  I made it down the next
>afternoon.
>
>>This isn't a crash, per se.  Is it some known cause, like a full disk
>>or like that-there?
>
>This isn't related to disk space.  In fact none of the clout "outages" for
>quite some time have had anything to do with disk space, regardless of
>misleading console messages.
>
>Here's a quick run down on the causes for a few of the recent events
>that lead to the need for local reboot:
>
>The most recent problem was with NIS.  The ypserv process got killed, and
>something wanted to check a NIS map, this will bring a Sun to it's knees.

Read:	This will indefinately hang the machine if ypserv dies the wrong way
	;-)  Unfortunately, ypserv can and does core dump once in a while.  
	This is why it is important to have TWO machines with ypserv running 
	(at a minimum) so the client can re-bind when it happens.

>Before that we had a remote reboot hang due to a memory error. This has
>happened about 3 times in the past few months.  We haven't traced the
>cause of our little memory parity error, but I suspect it might have
>something to do with line glitches, cosmic rays, heat, or malignant beings
>from the spirit world.

Or bad memory :-)

Seriously, do you have the RAM test enabled on the machine?  The PROM
default is to check only the first meg (not very useful!)  Then again, these
"ram tests" are frequently not useful anyway, but they are better than
nothing!

>We have altered the way clout handles incoming SMTP traffic, and
>sometimes heavy incoming traffic has overloaded the machine, on at least
>one ocasion this put the machine into an insane state, the console errors
>said something about being out of i-nodes however the problem was entirely
>a result of filled memory and swap space.

Filled virtual space will prevent the inode cache from being properly
managed, and can manifest as no available inodes.  Of course, you'll also be
getting complaints about other things (like no swap space), and processes 
will refuse to start (since there is no backing store or RAM for them 
available)

In the extreme case, this can completely lock you out - unless you get lucky
and something exits just as you're trying to log on.

>Now, during this time clout has had problems with spool space, including
>two instances of the mailbombing of a local client site.  NOT ONCE has
>this required a physical reboot of clout, and at least three times that I
>remember it hasn't even had any effect on the function of the clout
>machine.

It should not, IF you have parititioned your disks correctly.  It will,
however, cause mail to be dumped on the floor unceremoniously.

Anyone who runs a system with the mail spool on the root partition deserves 
what they get.  They will get it.  The pat answer to this is "don't do
that".

>In each case one or more members of the admin team have become aware of
>the problem early enough to head off any trouble.  And in the instance of
>the mail bombing when the client noticed that his filsystem had filled
>he contacted us and warned us well before things became critical on
>clout.

In general, there are several items which will hose you every time:

1)	Running out of virtual space (RAM + page).  This will, on occasion,
	crash systems (or it might just lock them up)  It is always bad news.
	
2)	Running out of disk space on root (frequently prevents anyone from
	signing on; ergo, you have a bitch of a time fixing the problem).
	Some machines also crash under this kind of stress.

3)	NIS problems can cause service failures or a completely locked up
	system (its not really locked, but system calls which attempt to do
	name <> uid mapping hang, so it appears locked).  NIS should NEVER
	be run unless you have TWO servers on EACH subnet.  NO EXCEPTIONS.
	Further, NIS should NEVER be run on a machine which is visible from
	"hostile" locations -- if it is, you're asking for someone to 
	download your password file and crack the passwords.  Filtering on 
	routers is only marginally useful in combatting this; even if you 
	prevent contact with the portmapper, people can still get through 
	by creative port guessing.  (This is less true for "secure" NIS, 
	but even that isn't very secure)  Run the Sun security (shadow 
	password and more) package instead, and get rid of NIS.

This is a good start.  CLOUT seems to be missing a few of these points ;-)

--
Karl Denninger (karl@ddsw1.MCS.COM, <well-connected>!ddsw1!karl)
Data Line: [+1 312 248-0900]