As you are aware, InterServer experienced minor network problems this past Monday, Tuesday and a major network problem on Wednesday. All service was restored late last night at our Secaucus Datacenter. This email explains the series of events that occurred, the resolution and what the final solution will be. Starting Monday afternoon we began to receive customer complains of network slowness only while using applications such as FTP or streaming media. We began to investigate by requesting source IPS, trace routes, ping test from the affected customers as most services were not affected at all. Although all these tests came out lean we saw a pattern forming as all trace routes came though our Core2-Teb router located in the first datacenter in Secaucus over Level3.
Starting Tuesday we began to run more tests on the core router, and as they all came back clean we began to suspect failed hardware. Our routers have 4 main parts: routing engine, dual power supplies, chassis, and fiber blades. Tuesday night we slowly replaced all fiber blades in Core2-Teb. The issue looked resolved and only caused minor outages to the primary or secondary redundant uplinks to clients edge routers. These replacements were done live and unnoticeable to clients. Wednesday morning our networking team started to actually see network issues due to decreased traffic levels and customers reported slow website loading times. Ultimately the situation with core2-teb had gotten worse so we still suspected hardware problem on core2-teb. The next replacement we did was power supplies in the router. Because they are redundant one can be taken out to replace without any interruption. Power supply causing these kind of problems while possible was unlikely. When that was complete the only thing left to replace then was chassis or Routing engine. It was a safe bet that chassis would never go bad since it does not have any working parts so we moved to the routing engine replacement. This is done easily because the routers we use are designed to support two routing engines – allowing the ability to install a second one, copy over configuration and failing over without interruption.
During the fail over process brief network slowness would have been noticeable for users going over Core2-teb while all traffic failed over to Core1-teb. Next we installed the old routing engine in a spare router and ran a full diagnostic test on it. Surely enough it did not pass every test and we were happy to see that this might have been the cause of the problem. Unfortunately the problem was still not resolved. After doing some more tests during the day with off and on network interruption we agreed that core2 needed a soft reset. We scheduled the reset for 6-7pm to avoid the networks busy time. Because the network is setup redundantly and we have tested the loss of a core router in the past with traffic failing over properly, resetting the router was thought to be a safe option. During the reset the router core1-lga (router in 111 8th ave Manhattan) had to handle a long series of BGP re-calculations to be able to reroute the traffic. During this wait our network was suffering from severe packet loss because customer routes were not being redirected to core1-jfk (router in 60 Hudson Manhattan) fast enough. After seeing the router was not making any progress we were forced us to disable parts of our network to take the load off core1-lga and then establish all the BGP sessions between our 5 core routers (3 in Manhattan and 3 in Secaucus) manually. BGP sessions with our providers also needed to be reestablished and after that customer traffic was enabled in small sections to slowly build the network back to its original level. During the router failure we noticed strange latency on core2-teb while no customer routes were being advertised there which was slowing down the bgp rebuilding process and caused router memory to run out on one BGP process. This lead us to investigate and find a bad link between core2-teb and core1-lga, of which there are 4 links. There were no errors on the link in the logs and it is disabled for repairs now.
We have looked into the series of events that caused the outage and will be implementing changes to ensure something like will never happen again. This year we had a planned a $120,000 network renovation that was going to take place early to mid 2008 which would allow us to support more bandwidth providers and larger BGP tables. This has now been moved up to December of this year. This upgrade will involve us replacing our routing engines to the latest sup720’s. These new routing engines have a faster CPU and support more memory than the current sup2’s that we run.
During a router failure this engine would be able to rebuild the BGP table in seconds as opposed to a 5 minute time. The next phase is activating the 6th, 7th, and 8th fiber we have going to NYC. This will allow us to support larger volumes of data and leave more than 50% room to fail over. We will also be upgrading the link we have between 1118th and 60 Hudson to 10GE instead of the current 2GE. The final phase will be to replace the remaining RiverStone edge routers with Cisco edge routers. This only affects users on RiverStone edge routers which is any one set up more than 2 years ago in datacenter Teb1. This upgrade will dramatically increase customers route performance and fail over time in case of lost connectivity to any of our Core routers.
Generated by Interserver, Inc