Network Maintenance Progress: Core Router Replacement [Expired]
Posted by Kevin Stange on 11 September 2010 11:39 PM |
|
11:00 PM: Routing loops have been resolved and routing should be normal. There may still be some isolated incidents, but if you do find one, please send in a ticket with a traceroute and we'll work on getting it resolved immediately. 10:00 PM: We are now getting intermittent OSPF routing loops causing intermittent inaccessibility and are working on getting those resolved. 7:20 PM: Core4 is actively being worked on and appears to be the last issue. Customers homed to Core4 only are down, but all other services should be normalized. If your account is still down, please contact our support department. 6:10 PM: There are off and on memory issues with one of the core routers still causing some issues that we are working with the vendor to resolve, but everything else should be online. If not, please contact our support department. 5:10 PM: Key components are now working, services should be coming up for most customers shortly. 2:29 PM: Restoration is taking longer than anticipated. ETA revised to 4:30PM CDT 2:29 PM: ETA is about 30m-1h. We'll fully update when things are back online, but it's expected we'll see a full restoration of connectivity before 3:30PM CDT 11:32 AM: Core routers are both online. We're bringing things back VLAN by VLAN; as soon as you're back online we'll email replies to existing tickets to confirm that your connectivity has been restored. 7:38 AM: We are continuing to work on outstanding issues caused by the router replacements and currently have no ETA on resolution. 6:06 AM: The problems being caused by the compatibility issue seem to have been solved by removing core4 and replacing it with the Brocade. We are still diagnosing some of the remaining issues and hope to have them resolved shortly. 5:15 AM: The issue was traced to a compatibility issue that prevents the Brocade and Cisco from working together and we are going ahead with completing the replacement of core4 at which point we will resolve all outstanding issues. We are expanding the maintenance period by one hour to deal with this problem. 4:58 AM: We are continuing to work on solving the problems caused by the new router with Brocade. We are attempting several possible solutions at this time and will report when we have additional information. 4:00 AM: We are currently working with Brocade support to attempt to resolve some of the issues that have resulted from the upgrade. We will provide an update as soon as we can. 3:23 AM: We are currently diagnosing known issues with packet loss that are affecting most customers. We will not proceed with the core4 replacement until this problem is resolved. 2:30 AM: core3 has been replaced and we are currently working on bringing the new Brocade router into normal operation status and solving a few complications from the change before we begin the replacement of core4. We will provide an update when this step is complete. 12:53 AM: We are now proceeding with the removal of core3 and replacement with its Brocade equivalent. We will provide an update if there are any issues with this process or when the replacement is completed. 12:25 AM: We are currently verifying configuration settings on the routers to reduce the risk of errors that may cause transition issues. We will provide an update when we are ready to begin the physical migration. 11:39 PM: The maintenance is on schedule to begin at 12:00 AM. Original MessageThis is a reminder of previously announced network maintenance which was scheduled on August 24th as per the following: https://support.steadfast.net/index.php?_m=news&_a=viewnews&newsid=284 This maintenance window begins at 12:00 AM US Central Time and completes at 6:00 AM. There will be various periodic service interruptions for all customers. We will provide status updates in this announcement visible on our announcements page. You can view the updates at the following URL: https://support.steadfast.net/index.php?_m=news&_a=viewnews&newsid=285 If you have any questions, please feel free to contact support as you usually would. Our web site, email and phone system will continue to be available during this maintenance period. | |
Comments (0)