Chicago network outage
UnknownUpdates
-
Investigating • We're investigating the Chicago network outage. We'll post updates here as we have them.
-
Investigating • This appears to limited to only some racks. We're continuing to investigate and will update this posting as we have more information.
-
Identified • It appears we've lost a power feed to some racks, which is affecting single PSU devices such as servers with a single PSU. Racks with server IDs srv14xx, srv12xx, srv13xx are affected. We are in communication with the building manager. We will post updates here as soon as we know more.
-
Identified • We've heard from other tenants in this building that they have been affected too. We remain in contact with the building manager. We will post updates here as we have them.
-
Identified • We are waiting for an update from the facility manager. We will share updates as soon as we have them.
-
Identified • Update from the building: "Please be advised on Thursday, April 24, 2025 at 9:45pm, the building experienced a loss of UPS power on the “A” side UPS distribution. While the B-side UPS distribution system in still online, Customers may have experienced a power loss on single corded equipment. We will continue to monitor critical systems until the issue can be resolved. Building engineers and electricians are working to determine the source of the outage and should be able to determine the root cause in the near future. Additional updates will follow for the duration of the event." We will post further updates as we have them.
-
Identified • This situation is also affecting servers with IDs srv10xx AND network ports starting with ge-1. Server IDs srv10xx and network ports starting with ge-0 are not affected.
-
Identified • Power is being restored and top of rack switches are booting. We're keeping an eye on things to make sure all racks come back online.
-
Unknown • This incident is resolved. If your server in any of the specified server ID ranges is inaccessible, please use the KVM console to make sure the server is powered on and booted into an operating system, and if you disabled your switch port to try to "fix" it that you've re-enabled your port, before opening a ticket. Tickets that can show that they've checked these things will be prioritized over other tickets.
-
Unknown • Update from the building on this situation: "As of April 25, 2025 at 12:45am, UPS, ATS and electricians have isolated the source of the outage to the “A” side UPS, and have now placed the A side UPS system in maintenance bypass. With the A side in bypass, the 717 data center is running with the B side UPS in normal operating mode, and the A side on stable utility power. The UPS vendor will continue to troubleshoot the issue with the A side UPS and determine recommended repairs. Additional updates will follow for the duration of the event. We will continue to monitor critical systems until the issue can be resolved." - will post further updates as we have them.
-
Unknown • Update from the building: "As of April 25, 2025 at 10:00am, the 717 data center continues running with the B side UPS in normal operating mode, and the A side on stable utility power. The UPS vendors (DC Group and Schnieder) will continue to troubleshoot the issue with the A side UPS and determine recommended repairs. Additional updates will follow for the duration of the event. We will continue to monitor critical systems until the issue can be resolved."