🚀 go-pugleaf

I am sure you will be aware that there are significant problems with
mail at the present time.

I can only apologise for the ongoing failure to provide timely email
delivery.

The following is some background on the problems to date.

On Friday last, the disk system on punt-1 started to report failures.
This is a RAID array so such problems do not immediately lead to loss of
data, however in order to secure the data, punt-1 was removed from service
and the data copied to a new machine.

This new machine was part of the planned improvement to the mail system
and brought into service ahead of the implementation plan. It is a
machine that is intended to be used as a networked disk array to hold
users mail.

Copying the data was completed early on Saturday and the machine punt-1
was reconfigured with good disks to come back on line during Saturday
morning.

Unfortunately it was discovered that the process that notifies the
punt machines that a user is connected was failing to notify punt-1
correctly of this information. This accounted for the failure of punt-1
to deliver mail at the weekend. Some considerable effort was spent
diagnosing and attempting to rectify the problem. At several times it
was believed to have been cured, only for the problem to re-occur.

During Saturday night / Sunday a replacement more powerful replacement
machine was built to replace the existing punt-1. This was brought into
service, but failed to be effective in curing the problem.

As time progressed a further problem was discovered. The 'locks'
that the system placed on customer mail during attempts to deliver
were not being cleared. Thus later attempts to deliver mail were
starting correctly, but failing when the punt discovered that the mail
was locked. This accounts for the connections made to customers when no
mail was transferred.

The current situation is that punt-1 is being rebuilt again, this time
as a 'clone' of punt-3 which is the most effective of the mail machines.
In order to do this, it has been necessary to withdraw punt-3 from
service for a short period.

The resolution of this problem is being given the highest priority by
the Operations staff at Demon. At this stage it would be somewhat
foolhardy to offer a guaranteed resolution time, but we are endeavoring
to fix the existing problem by this evening.

We have been asked why we don't move the mail from punt-1 to another
machine. Doing so would undoubtedly overload the other machines and
reduce service even further. We are not prepared to take that risk.

It may be of interest to note that the Demon mail service is currently
handling an estimated 8,000,000 messages per day.

I can report that no mail has been lost. However, the overloading of
the demon machines has meant that some mail will be backlogged on remote
machines.  Once delivery starts to run properly again, this overloading
will drop and so mail will get transferred to Demon, and hence to our
customers.

I apologise for the severe inconvenience you are experiencing.

Malcolm Muir
Demon Internet

-- 
Malcolm S. Muir                                Demon Internet Ltd.
Sunderland                                     322 Regents Park Road
England                                        London N3 2QQ