Thread: Prime Dallas DC - Power Outage affecting multiple providers

Originally Posted by hermetek

Yeah, we were back up for a couple hours and are now down again.

Back up again. Fortunately it looks like it was a short network issue this time, not power to our equipment.

Originally Posted by Zhenmue

can anyone confirm if everything offline again?

The servers I am monitoring look online from here.

We are still down at TMS. Waiting for update....

Originally Posted by MechanicWeb-shoss

The servers I am monitoring look online from here.

was network it seems, and took 1-2 minutes, but for all servers.

Hostrush is also still down.

Originally Posted by Zhenmue

can anyone confirm if everything offline again?

Originally Posted by hermetek

Yeah, we were back up for a couple hours and are now down again.

I don't know who your providers are, but FYI FWIW that Watching Grass Grow is hosted at TMS and has been up continuously since the extended outage that hit the entire DC this morning.

Originally Posted by TotalChoice - Bill

We are still down at TMS. Waiting for update....

Sorry you are still down Bill - perhaps you are in an area that was impacted more by the outage?
If you haven't already, maybe worth opening a ticket at TMS (Jose has always been very responsive, although I'm sure he's super busy right now) as maybe possible you have an outlier that may need a "nudge"

Originally Posted by alekk

I don't know who your providers are, but FYI FWIW that Watching Grass Grow is hosted at TMS and has been up continuously since the extended outage that hit the entire DC this morning.

We're with Psychz. Our BGP peering with them dropped for 3min about 45min ago.

Originally Posted by hivelocitygm

they should be replaced and online within the hour.

Unfortunately an hour later and both of these Hivelocity servers are still down:

Facility Row AN
Position
22
Rack
Rack 71 (Facility Rack 25)


Facility Row AN
Position
44
Rack
Rack 60 (Facility Rack 14)

Looks like there are more updates with actual information on the LET forum than on WHT:


https://lowendtalk.com/discussion/19...-power-outage/

Just wanted to post an update that we are still down over here at Hivelocity due to failed switches.

8 hours and counting... It's been unfortunately not easy getting communication from Hivelocity. Our latest update is that engineers are being flown in to Dallas to help later tonight. No ETA as to when to expect a resolution.

I have a dedicated server that is hosted in the trouble data center in Dallas. The last thing I would expect with something like a data center. Anyway, seems like my server is still offline, and no idea if I have my data still intact. This might end up becoming a seriously complicated problem given how this data center seems to be hosting services for multiple kinds of clients, service providers, and so forth.
Psychz dot net change logs showcase a very vague statement about the issue sent out at 7:18 Dallas time.

Code:

A severe storm impacted the Dallas area, causing a significant disruption in the power supply to our data center. This incident led to the complete loss of data center operations. Our engineers are working with the local data center to get everything back online.

POP: DALLAS
DATE: May 26, 2024
ETA: it might be down the entire day

A good experience though to go through with a personal server I use just for my hobby projects. Fingers crossed it will be back up soon. Was out for me today as of 7:00 AM Dallas time. Be safe all.

Originally Posted by trvdg

Just wanted to post an update that we are still down over here at Hivelocity due to failed switches.

8 hours and counting... It's been unfortunately not easy getting communication from Hivelocity. Our latest update is that engineers are being flown in to Dallas to help later tonight. No ETA as to when to expect a resolution.

Hopefully they can get you back online soon!

I know we had a switch go down and even a line card in one of the router acting up but it's all redundant other than one customer on that port as a backup bandwidth port.
Servers and equipment hate these types of power outages it can cause failures. It's our #1 cause of failures we have seen over the past 20 years dealing with tech.

Communication is always key in these types of outage.

Originally Posted by Purevoltage

Hopefully they can get you back online soon!

I know we had a switch go down and even a line card in one of the router acting up but it's all redundant other than one customer on that port as a backup bandwidth port.
Servers and equipment hate these types of power outages it can cause failures. It's our #1 cause of failures we have seen over the past 20 years dealing with tech.

Communication is always key in these types of outage.

Unfortunately the last update we have is from an hour ago promising a resolution "shortly" which doesn't tell us much...

"From my understanding our team is working on this rack now and should have functionality shortly."

Back online now. I was able to look at the current uptime of my server, shows that it has been running since 8:59 AM this morning DALLAS TIME, which is interesting since I only just have been able to reach it. Seems like the power outage was mainly an issue that affected the external network access for the data center, which might indicate that the internet service reinitializing from an unplanned shutdown following an energy grid reset, may have been the primary issue, rather than a general power outage.

This would indicate that any kind of cause attributed to "an act of god" cause may not be justifiable... Given..., it seems to have been a localized issue related to external network access, likely stemming from internet service disruptions following an energy grid reset. Just my two cents provided for those who might need to write a response report.

Originally Posted by Purevoltage

Communication is always key in these types of outage.

I am really disappointed in how some of the popular server providers handled this outage. Outages can happen; we all understand that. But a lack of communication is bad.

Some providers' websites, along with their support systems, went offline and remained offline until client servers started to come back online. That's 2-3 hours with zero communication. Some merely mentioned there had been an outage on their status pages with no other updates. Some providers' status pages say everything is running fine, but end users still have servers offline. And many didn't even bother to come to communities to talk to end users.

This is probably as bad a situation can get.

Originally Posted by trvdg

Just wanted to post an update that we are still down over here at Hivelocity due to failed switches.

8 hours and counting... It's been unfortunately not easy getting communication from Hivelocity. Our latest update is that engineers are being flown in to Dallas to help later tonight. No ETA as to when to expect a resolution.

My hivelocity server is now online

time for SLA requests?

This is what happens when folks never test their backup power. You can have all the generators and UPS in the world, but if you never test it, this is exactly what you get. Isn't the first time and certainly won't be the last. Really makes me question this datacenter though.

Originally Posted by wswd

This is what happens when folks never test their backup power. You can have all the generators and UPS in the world, but if you never test it, this is exactly what you get. Isn't the first time and certainly won't be the last. Really makes me question this datacenter though.

I have to agree with this. We had this happen in another DC in the past. Happened twice in a month then it's been fixed since then.
It's hard to trust any place to be honest it's why we tell everyone to always have backups offsite for anything critical.

Just sucks these issues have to come up on long weekends most times as well.

Unfortunately, another "1 hour" promised timeline has come and gone.

If Hivelocity would've simply said this morning, "This is a major issue. We expect a 12 hour downtime" we would've taken more drastic measures. Instead, we've been strung along all day with "1 hour" promises that fail to materialize.

One hour ago I was promised a phone call or ticket update if the resolution was going to take more than 1 hour. Nothing. No update.

Originally Posted by trvdg

Unfortunately, another "1 hour" promised timeline has come and gone.

If Hivelocity would've simply said this morning, "This is a major issue. We expect a 12 hour downtime" we would've taken more drastic measures. Instead, we've been strung along all day with "1 hour" promises that fail to materialize.

One hour ago I was promised a phone call or ticket update if the resolution was going to take more than 1 hour. Nothing. No update.

Please check your server now. If you are still not able to reach your server please update the existing support ticket. You have my sincerest apologies that you have been dealing with this all day.

My servers @ Webhostpython are still down...it's going to be 24 hours of downtime soon at this rate, have been lucky over the last few years to not experience anything like this bad. This shows that the Dallas Datacenter is not serious. The amount of time it's getting to fix this is too much.

Originally Posted by hermetek

We're with Psychz. Our BGP peering with them dropped for 3min about 45min ago.

Are your servers back online with Psychz? Mine is still down, I have some servers in Dallas that is already back online but some are still down and the weird thing is, I don't know if its related to the storm in Dallas but I also have some servers in Los Angeles that also went down.

Originally Posted by HypeKoder

Are your servers back online with Psychz? Mine is still down, I have some servers in Dallas that is already back online but some are still down and the weird thing is, I don't know if its related to the storm in Dallas but I also have some servers in Los Angeles that also went down.

If it is on multiple locations it could be something else, get in contact with them. Our cabinet with Psychz was down for 2 hrs 50 mnts. And then a brief 4-minute downtime possibly because of a network equipment reboot at dc.

Originally Posted by wasiflaeeq

If it is on multiple locations it could be something else, get in contact with them. Our cabinet with Psychz was down for 2 hrs 50 mnts. And then a brief 4-minute downtime possibly because of a network equipment reboot at dc.

Now this is f**ked up,

In order to investigate, we checked the server console and did not find any OS on the server. Please find the attachment for your reference.
Please let me know if you need to install any OS and need assistance to do so.

I have backups and everything but this just sucks, I wonder if this happened on my other servers too as I still have multiple servers offline in Dallas.
You are lucky your servers was down for like 3 hrs only while some of my servers was down for 8 hrs and I still have like 3 to 5 servers that are still offline.