Thread: Prime Dallas DC - Power Outage affecting multiple providers

Originally Posted by Hannan

Do you have any ETA? Our servers are still down

I would suggest you open a support ticket so we can take a look at your specific server(s).

Originally Posted by selinux

The part where the storm caused destruction allover the DFW area, homes roofs being rip off, electricity going out, and so much more.

But not the DC, as it seemed he was alluding to.

I would like to see an post-event explanation from Prime and a remediation plan for how they're going to prevent future outages of this nature. DCs really shouldn't go down like this.

Originally Posted by bear

I must have missed where it was explained as a natural disaster?
It's my understanding the power went out and the generators weren't passing power back to the equipment.

What did I miss?

You are probably right.

I am just making a best guess based on what I saw on TV/YT.

No provider on this thread said what caused the outage in the first place that led to backup UPS not getting power or that caused starting the generators.

Two of my Hivelocity servers are still down. They are powered on, but have no network access, even local. IPMI works though.

I opened a support ticket and just called and the support guy said, "Call back or open a support ticket in 30 minutes if they aren't back online." Didn't take any information or ask which servers or anything.

The power came back 30 minutes ago and most servers came back online right after that. Unfortunately these are critical ones.

It's been almost four hours now.

This is my support ticket ID: #1176036

Originally Posted by Hannan

Do you have any ETA? Our servers are still down

We are online now

Originally Posted by trvdg

Two of my Hivelocity servers are still down. They are powered on, but have no network access, even local. IPMI works though.

I opened a support ticket and just called and the support guy said, "Call back or open a support ticket in 30 minutes if they aren't back online." Didn't take any information or ask which servers or anything.

The power came back 30 minutes ago and most servers came back online right after that. Unfortunately these are critical ones.

It's been almost four hours now.

This is my support ticket ID: #1176036

I have a technician checking on that ticket now.

Originally Posted by hermetek

All my Psychz equipment is back online and has been for the last 1 hour 14 minutes.

I'm still entirely down at Psychz!

Originally Posted by hivelocitygm

I have a technician checking on that ticket now.

Here is the response I've gotten:

"In regards to an update our team is slowly bringing everything online and replacing any top of rack switches that are experiencing issues.

I do see for device #27811 that IPMI is not pinging likely meaning were going to need our team to physically inspect the switch.

Additionally for device #17059 I do see its booted into the OS with no connectivity. That switch is missing its virtual chassis.

Our networking team has been notified and they are reviewing the switch now. I will look to provide more update for device #27811 and will update you as soon as possible."

How long is that kind of work expected to take? It's been over 4 hours now...

My HostUS VPS still seems to be down. I'm assuming it's part of this outage.

sadly, we also have some servers that remains Offline on that location.

We also have an online server in hivelocity but without network connectivity.

Originally Posted by 4ServNet

I'm still entirely down at Psychz!

As are we. We have a full cab with A+B power and VRRP. Really poor handling. They have our ticket on "Hold" status and say they "aren't going to do anything until they receive the greenlight" from the DC. This isn't an issue of waiting on jurisdictional clearance due to active line down. They don't have appropriate staffing and is completely unacceptable.

Apparently Hivelocity is having issues with switches not rebooting or something and needing to be replaced.

I don't understand how that's related to a power outage, but so far it's 5 hours of downtime for us. Still no ETA.

Originally Posted by trvdg

Apparently Hivelocity is having issues with switches not rebooting or something and needing to be replaced.

I don't understand how that's related to a power outage, but so far it's 5 hours of downtime for us. Still no ETA.

Switches are just like servers in that they do not like suddenly losing power. Sometimes they come back up and sometimes not. We have spare switches and backups of all switch configurations so it is just a matter of time for us to get any failed switches replaced and back online. Power has been restored to 100% of our cabinets and we are now just working through support tickets addressing individual servers and switches with issues. We have techs on-site in Dallas working through all of this as quickly as possible.

If you have a ticket opened please provide me that ticket number so I can have someone jump on it.

Originally Posted by trvdg

Apparently Hivelocity is having issues with switches not rebooting or something and needing to be replaced.

It seems their status page says there are no issues:


https://status.hivelocity.net/
https://status.hivelocity.net/785705204


Perhaps folks should make this a wake-up call to the server providers and data centers. If you have servers with providers that did not provide timely updates via their status pages or support systems, perhaps it is time to move critical systems away from them. To clarify, I am not referring to Hivelocity (many providers have been affected). They did post periodic updates on this thread, though the status page could have been better.

Seems still many of them are not online yet.

Originally Posted by MechanicWeb-shoss

It seems their status page says there are no issues:


https://status.hivelocity.net/
https://status.hivelocity.net/785705204


Perhaps folks should make this a wake-up call to the server providers and data centers. If you have servers with providers that did not provide timely updates via their status pages or support systems, perhaps it is time to move critical systems away from them. To clarify, I am not referring to Hivelocity (many providers have been affected). They did post periodic updates on this thread, though the status page could have been better.

This page monitors the routers at each location hence the header of that page "Hivelocity Routers". You can see the page does indicate downtime today in Dallas. That page is not reporting the status of the top of rack switch within each individual cabinet which is what the customer is having an issue with.

Something new?

All of our servers with Hivelocity back online almost 2 hours ago. For two servers, human intervention was required. After opening ticket, they take care the issue almost immediately.

TMS servers also back online now. One server required human intervention.

Originally Posted by hivelocitygm

Switches are just like servers in that they do not like suddenly losing power. Sometimes they come back up and sometimes not. We have spare switches and backups of all switch configurations so it is just a matter of time for us to get any failed switches replaced and back online. Power has been restored to 100% of our cabinets and we are now just working through support tickets addressing individual servers and switches with issues. We have techs on-site in Dallas working through all of this as quickly as possible.

If you have a ticket opened please provide me that ticket number so I can have someone jump on it.

5 hours and 40 minutes of downtime so far for us. Two of our servers, critical ones, are having the issue with failed switches.

Ticket ID is 1176036. I've been told "all the switches in downed racks are being replaced" but again I have no idea how long that is going to take. Are there hundreds? Will this be five more hours? 1 hour?

Originally Posted by trvdg

5 hours and 40 minutes of downtime so far for us. Two of our servers, critical ones, are having the issue with failed switches.

Ticket ID is 1176036. I've been told "all the switches in downed racks are being replaced" but again I have no idea how long that is going to take. Are there hundreds? Will this be five more hours? 1 hour?

they should be replaced and online within the hour.

can anyone confirm if everything offline again?

Originally Posted by Zhenmue

can anyone confirm if everything offline again?

Yeah, we were back up for a couple hours and are now down again.

Originally Posted by hivelocitygm

This page monitors the routers at each location hence the header of that page "Hivelocity Routers". You can see the page does indicate downtime today in Dallas. That page is not reporting the status of the top of rack switch within each individual cabinet which is what the customer is having an issue with.

Thanks for the clarification.

This was not actually about you. I can see you are posting updates, and users are also saying you are responding to support tickets. That's what is called communication. I appreciate that.

If I have to say anything about the status page, I appreciate that it shows downtime. And there was no information about the downtime on the status page. I also get that it shows the uptime status of the router, not the cabinets. But this was a data center-wide issue. The status could have indications that all systems are not yet operational.

Some of the affected providers didn't post anything on their status pages/support systems or post any (or hardly) updates at all. I was referring to these providers, excluding HVLCTY.