🚀 go-pugleaf

As many of you noticed clout.chi.il.us was dead most of the weekend.  The
cause of this was a rather complex combination of factors, each alone
almost but not quite enough to mess things up.  The combination,
catastrophic.

First off, sometime Friday someone managed to fill up our very limited
spool space.  This caused clout to refuse connections (even for people to
pick up mail...) and to generate a whole bunch of spurious bounce messages
for mail that was later delivered.  If you saw bounce messages with an
error saying that uux failed there is a good chance that the mail actually
got through eventually.  If you recieved a bounce of this nature for
anything important, you may want to check and see if your mail did arrive.

Later Friday evening there was a problem with clout's ethernet connection
and a few router machines that connect clout to the rest of the world.
For some reason clout lost the ability to talk to all but one thing on
its ethernet, and even that link was somewhat messed up.

So, late Friday night clout was rebooted.  Shortly after this the machine
experienced a memory error, and lock up on the SCSI bus that panic'd the
machine.  It was down all the way.

The admins were unable to get in to reset the machine until late Sunday
evening, early Monday morning (about 12:15 AM).  Things seem to have been
working fine ever since then.

Even though clout was dead most of the weekend, most mail for chi.il.us
sites did not bounce.  There are secondary nameservers for the chi.il.us
domain that took over for clout when it died.  And all sites in the
chi.il.us domain have secondary MX hosts which accepted mail for them
until clout resumed operation.

So, with a very few exceptions no mail should have been lost, and no mail
for chi.il.us sites should have bounced as a result of clout having been
dead.  The only net effect for most sites will be the 2 day delay in
delivery.  A very few sites might have lost some mail as a result of an
outdated configuration entry in the nameserver database causing a delivery
addressing conflict.  This is now known about and is being addressed.

If you have any questions, or comments, feel free to contact either
myself, or one of the other clout admins.
--
Russell Rezaian			|  rrezaian@austral.chi.il.us
				|  rrezaian@zed.chi.il.us
				|  rrezaian@earth.eecs.uic.edu