The chubby gnomes should come again!

I thought you guys might be interested in this email from the CEO of our ISP. I don't want to name them, but I'm sure you can figure out who they are.

---------

Hello Matt,

Well, we had quite a wild ride last night.

Unfortunately, it wasn't the type of ride we paricularly enjoyed.

At 1:16AM PST, our link to AboveNet crashed. (We also have links to Peer1 and InfoRelay, which did not go down and have remained online this entire time.)

After examining our switch, router, and fiber port that were linked to AboveNet, we could not find anything wrong, so we called AboveNet's 24x7 NOC.

The next 11 1/2 hours were mainly a blur to our weary support team, but we will sum it up quickly as follows:

* We had a complete "carbon copy" of our networking equipment as spares. We plugged it all in. It did not work.

* AboveNet insisted it was our fiber module (also known as a SFP) that had died. We spent hours on the phone with Cisco TAC and finally sourced a new one for $2200. (Mind you, we had two at the datacenter that did not work, but AboveNet had convinced us that that was the problem.) We received the new SFP via express courier at 8:30AM. We plugged it in. It did not work.

* More hours of phone support with AboveNet, who reset our switch configuration (which was working just fine before the link dropped -- we had changed nothing on our end) and tried many hacks on our end to get the link to work. No go. We also had a Cisco-certified tech called in to look at our configuration. He said it was fine.

* At 11:30AM, we received a phone call from AboveNet that they were sending out a tech to investigate in person. At around 12:15PM, he arrived. He examined our switch configuration and immediately asssessed that our end was fine. He left and went to AboveNet's SJC2 datacenter (where the other end of our fiber terminates.)

* At 12:45PM, he called and said that our link to AboveNet had been plugged in to a switch port on AboveNet's end that had died. He changed us to another switch port (on a different switch) and we were online at 12:57PM PST.

Essentially, it took AboveNet 11+ hours to figure out the problem was on their end after all, was not our fault and, in fact, had nothing to do with our specific configuration -- it was just a bad port on their end.

We will be making quick, firm, and decisive changes to our infrastructure to alleviate this problem in the future:

* We will be re-IPing, as we have mentioned earlier. All of your IP addresses will change. This is to get us out of AboveNet's grip and get us OFF of their network as quickly as possible. We will be expediting this IP address change and will do it sooner than originally planned to alleviate further unplanned downtime. You will see more emails about this by the end of this month. (Customers reading this who are being moved to Empowering Media's servers due to the buyout of 10for10.com -- your IPs will be changed separately; please pay attention to emails from Empowering Media about this.)

* Once we re-IP, if ANY carrier (AboveNet/Peer1/InfoRelay) goes down, our network connection remains up, since our IP addresses are no longer tied to our upstream provider. In a matter of a few seconds, if AboveNet goes down, your connection will AUTOMATICALLY fail over to our other providers. This is why re-IPing is so critical.

* We will be pursuing serious financial compensation and damages from AboveNet. We may pursue legal action if it is determined that it is necessary in order to have damages awarded.

* As soon as all of our customers have re-IPed, we will be dropping AboveNet as a carrier due to their negligence.

Make no mistake. This has cost us dearly in terms of reputation, time, money, and customers. While I can't say I blame you for being angry, as I too am angry, please do communicate with us and we will do our best to help you in whatever way we can. Assuming AboveNet does decide to award us damages, we will be in turn sending this compensation on to our customers. If AboveNet does not award us damages, or a drawn-out court battle ensues, we will do our absolute best (within our power) to help you. Since this outage was not directly our fault, we will need to see how things shake out with AboveNet before announcing any particular financial compensation. While I cannot promise you that we will be able to compensate you for this downtime, I can promise you that we will be fighting with every breath we have to make sure that this is the last outage like this we see, and that we will prove once again our willingness to go the extra mile with our customer service and make sure that you are satisfied with our service.

Will this happen again? After we re-IP, no, it won't. That's why re-IPing will become our top priority over the next 30 days. You have my promise -- my word -- that we will do everything in our power to prevent any outages like this from happening again.

---------------

I'm not really interested in compensation from them (I don't think they owe us that really), though free hosting next month would be cool.

--matt

Has anyone else noticed how laggy it's been these last few hours? Not every so often lag - just lag on everything.

Hey, I know Matt was talking about changing ISPs, and...well, if he did it about 5 days ago...can we go back? Lag bad.

That whip crack is going to turn on far too many people.

If you get turned on by a smilie, I worry about you people......

If you get turned on by a smilie, I worry about you people......