LA Utility Power Partial Outage Saturday 30th November 2024 04:43:00

We're aware of an issue impacting some servers with IDs beginning with srv3xxx, and we are investigating now.

If your server ID begins with srv3xxx, it's due to this issue.

Original information from CoreSite LA2 Team: Description: CoreSite field operations has confirmed we experienced a power quality event at the data center. Due to the utility voltage inconsistency, Our Generators did turn on. Our local Data Center operations team is currently investigating further on the cause and will provide further updates as we receive them. You may direct any questions or concerns to the below department:

Latest update from Coresite LA2 Team: Site has return to normal Utility Source with no Data Center Operations intervention. DCO team is currently continuing site checks.

We will update this posting as we have more information.

Update: One of our technicians has been deployed and is currently en route to address the situation.

Update from Coresite LA2. At this time we too are going to close this incident.

Update: Data Center Operations has confirmed all systems remain stable in their current configuration after our last PQE. All UPS systems are on inverter, with battery back up available should it be required. UPS 1B is still charging batteries and will require at least another 12 hours to reach full charge. All building load is on utility power with the exception of Mezzanine 1 (CR185), the Mezzanine remains on the emergency source generators awaiting arrival from our support vendor to troubleshoot our transfer device, estimated time of arrival is 4:00 pm pacific. We will be closing this incident communication at this time and focusing our communications to those clients located in the Mezzanine who remain supported by our generators.

Posted 2 weeks ago by dedicated

Update from Coresite LA2:

Update: We have experienced another PQE at the facility. The site is performing rounds to confirm all systems are operating properly. UPS 1B is back on inverter and all PDUs are restored.

Posted 2 weeks ago by dedicated

Update from Coresite LA2:

Update: Data Center Operations needs to restore a PDU breaker downstream of UPS 1B. In order to do this, we will need to place UPS 1B back into bypass. Once the PDU is restored, we will place UPS 1B back on inverter, we expect this evolution to be 20 minutes.

Posted 2 weeks ago by dedicated

Update from Coresite LA2:

Update: Data Center Operations has placed UPS 1B back on inverter, all UPS systems are supported by battery back up presently. Mezz1 remains on the emergency source generators with a vendor dispatched, arriving later today, to help troubleshoot the transfer device for this system. All other spaces are supported by the normal source utility.

Posted 2 weeks ago by dedicated

Update from Coresite LA2:

Update: Data Center Operations is reporting a subsequent PQE has occurred at the facility. The team is currently verifying all systems for proper functionality.

Description: CoreSite field operations has confirmed we experienced a power quality event at the data center. Due to the utility voltage inconsistency, Our Generators did turn on. Our local Data Center operations team is currently investigating further on the cause and will provide further updates as we receive them.

Posted 2 weeks ago by dedicated

Update from Coresite LA2:

Update: Data Center Operations confirms only Mezz1 remains supported by the emergency source generators, all other spaces are supported by normal source utility. our UPS vendor has arrived onsite and is working to put the UPS 1B back on inverter, all other UPS systems are in normal. We have also dispatched an additional vendor to help troubleshoot our transfer system.

Posted 2 weeks ago by dedicated

Update from Coresite LA2:

Update: Data Center Operations is confirming both Mezz1 and Mezz 2 are being supported by the emergency source generators, all other spaces are supported by normal source utility. We have brought UPS1B system modules back online and are charging batteries while the system remains in bypass, all other UPS systems are in normal.

Posted 3 weeks ago by dedicated

Update from Coresite LA2:

===. Update: Data Center Operations has confirmed all systems are online downstream of UPS1B however, this load is on bypass without battery backup availability. ===.

This means that all single-PSU servers should likewise be on now. If finding your server inaccessible, please try to power it on in SynergyCP if not already powered on (just need to power it on once; trying to power it on many times won't make it power on if the first time didn't work), and check the KVM console to see if the server is booted into the operating system, before opening a support ticket.

We'll post another update when UPS1B is re-engaged.

Posted 3 weeks ago by dedicated

Update from Coresite LA2 below. This means single PSU servers should be coming online shortly.

Update: Data Center Operations has confirmed all site load is back on normal source utility power. We have also successfully taken UPS1B system into maintenance bypass and are working to restore downstream breakers one at a time to reduce inrush.

Posted 3 weeks ago by dedicated

Update from Coresite LA2:

Update: Data Center Operations team will need to back out of Generator and will place site back to Utility source. Our Data Center Operations team has found an issue upstream with Generator transfer switch. Site will remain as it was prior while the team works placing UPS 1B in to maintenance bypass to troubleshoot the issues.

Posted 3 weeks ago by dedicated

This incident is mostly resolved. Coresite continues to work on A+B power distribution issues. This means that some servers may be offline if they are single PSU. These can include:

SOME E3-1270 servers that do not have "(mc)" or "(vlp)" in their SynergyCP CPU name. SOME E3-1650v3/v4. SOME Ryzen 5900X. SOME colocation with single PSU, or dual PSUs with a dead PSU.

These coming back online are dependent on Coresite restoring full A+B feeds. We will update this status posting when we have been notified of power stability.

Posted 3 weeks ago by dedicated

Update from Coresite LA2: Update: Site is on generator power while our team continues to troubleshoot issue taking place at the moment.

Posted 3 weeks ago by dedicated

CoreSite Update: Please be advise our DCO team has found that there is a partial outage on our PDUs that your power is on. Our team will be workign on placing the unit into maintenance bypass and get a vendor out to help with troubleshooting efforts. We will provide further updates as we receive them.

Dedicated.com Translation: The PDUs mentioned are not our cabinet-specific PDUs connected to our equipment. Instead, these are the facility's PDUs that supply power to our cabinets, which explains why some cabinets have not fully recovered.

Actionable items: We have identified that some of our cabinets are operating on a single surviving power leg. Our technician is already onsite reviewing the situation and, if necessary, relocating single PSU devices (primarily customer colocation equipment) to the surviving power legs.

Unfortunately, the power cutovers from CoreSite were not smooth and resulted in some outages. Our network devices are confirmed to be online, and the remaining outages are limited to single PSU devices on the unrecovered feeds. We are actively working to move these devices to the surviving leg where possible.

We will provide further updates as the situation evolves. Thank you for your patience and understanding.

Posted 3 weeks ago by dedicated

CoreSite Update: Data Center Operations team found power feed PDUs in LL partially offline. DCO team is currently reaching out to Vendor for a dispatch to assist in troubleshooting. We will provide ETA of vendor dispatch once received.

Dedicated.com Translation: These are not to be confused with our own cabinet PDUs connected to equipment in our cabinets, the mentioned PDUs feed our cabinets, this explains why some of our cabinets have not fully recovered.

Posted 3 weeks ago by dedicated