🚀 go-pugleaf

First of all, apologies for the late posting of this report. I was
the only person with "all the facts" (whatever they may be) and I have
been tied up on some other immediate priority work.

US Connectivity Outage - Last Saturday
--------------------------------------

At almost exactly 05:00 our time on Saturday morning we lost both
operational T1 lines to the US. This problem was reported
AGIS/Sprint/Mercury and the cause was the extereme weather conditions
washing out lines on the East Coast of the US.

We were unable to get a time to fix, and it transpired through the
day that the break itself was in a Wiltel service that is used by
both companies to deliver traffic from New York to Washington. This
break was reported to be under 6 feet of freezing flood water,
which was not helping matters.

To cut a very long story, and a long day, short, the Sprint service
showed signs of recovery at about 22:45 through being diverted by
an alternative land route in the US and was entually judged stable
enough to be restored to service about 0100 Sunday morning. The
AGIS line was recovered not long after this.

The scale of the problem in the US was described as affecting about
130 DS-3 circuits. FYI: A DS-3 circuit is 45Mbs/sec (about 30 each
of the T1's we have). Wiltel had problems trying to reroute that
amount of traffic we were told.

AGIS also had their Wiltel circuit running via a bridge in Penyslvania
that got washed out later in the say, which affected the speed of
recovery of the AGIS T1.

Connectivity to the UK via the LINX and parts of Holland (via our
peering arrangement wuth NLnet) was not affected. Customers may have
experienced problem reaching these sites when trying to resolve
system names via the DNS owing to the lack of connectivity to the
root name servers. This was worked around for customers using the
Demon nameservers by about 12:30 on Saturday.

Regards,
--
Peter Galbavy                                                 peter@demon.net
@ Demon Internet                                      phone://44/181/371_3700
                                            http://www.wonderland.org/~peter/
                  snail://UK/N3_1TT/London/42_Hendon_Lane/Demon_Internet_Ltd/