Thread: Prime Dallas DC - Power Outage affecting multiple providers

The storms/tornadoes hit Saturday night. There was only a passing thunderstorm this morning in the area of the datacenter. If a tornado had caused damage, I think it would've been offline overnight.

Unfortunately some of the updates above are correct. The Datacenter is having a power issue where the UPS' are not receiving power so everything that's supposed to have backup power (ie. All servers and equipment) is off. They're working with utilities and vendors to get everything restored ASAP.

No storm damage here as far as I can tell, so I don't believe it is related to that. I don't have details as far as the cause.

Everyone is here working on the issue, and more and more people arriving by the minute. We'll get back online as soon as humanly possible.

I have a server with VersaWeb they are down too because of power outage.

Originally Posted by TMS - JoseQ

Everyone is here working on the issue

Thanks for the update, Jose. At least we know the issue now.

Originally Posted by bdwebservices

I have a server with VersaWeb they are down too because of power outage.

Everyone affected who use prime datacenter.

Indeed, they have just told us that the entire data center is off, we hope it can be resolved soon.

Our Dallas facility provider, Prime(formerly Evocative and Carrier-1), has lost utility power. Initially the power load was transferred to battery backup and then generator as designed. However, at some point in time the generators dropped the critical load. Prime has electrical engineers investigating the problem and the solution. We will update you all as things progress.

Can a mod change this thread title to "Prime Dallas DC - Power Outage - TMS, Hivelocity, etc

Originally Posted by SecureZone

Can a mod change this thread title to "Prime Dallas DC - Power Outage - TMS, Hivelocity, etc

Psychz is probably worth mentioning too.

Originally Posted by TMS - JoseQ

The Datacenter is having a power issue where the UPS' are not receiving power

Originally Posted by hivelocitygm

Our Dallas facility provider, Prime(formerly Evocative and Carrier-1), has lost utility power.

Any idea what caused them to lose power? Storm?

Yeah they have lost both A+B power due to poor design it seems.

Over 2 hours now.... ugh.

Originally Posted by TMS - JoseQ

Unfortunately some of the updates above are correct. The Datacenter is having a power issue where the UPS' are not receiving power so everything that's supposed to have backup power (ie. All servers and equipment) is off. They're working with utilities and vendors to get everything restored ASAP.

No storm damage here as far as I can tell, so I don't believe it is related to that. I don't have details as far as the cause.

Everyone is here working on the issue, and more and more people arriving by the minute. We'll get back online as soon as humanly possible.

This just crossed the 120min mark for us. How bad does it look out there. Might have to start backup restores soon….

Should restore backups asap to be honest to be safe as that can take sometime to do.
No ETA given to us as of yet.

Originally Posted by IndicSD

This just crossed the 120min mark for us. How bad does it look out there. Might have to start backup restores soon….

Thread title edited to reflect the larger issue.

Ditto that appreciate the updates Jose - sounds like a really strange one, so once the dust clears (and you are able to have a beer or two!), be curious what happened.

Good call Bear to update the thread title to better reflect the wide-spread outage.

Psychz just posted an update in their portal that says "ETA: it might be down the entire day"

If we are down the entire day due to misconfigured backup power we're moving our entire infrastructure. That's insane.

Wow, glad I activated a partial backup... better than nothing! Good luck to those working on this!

I have been in webhosting since 2006... anytime I have seen a DC go black like this, when it was the whole DC, it has taken at a minimum 12 hours to resolve, and sometimes days, and a few times with multiple ups and downs of power while they try to fully resolve the issue... I am not saying that is going to be the case here, but I am already working on restoring critical systems and clients to VM's at other DCs.

It has been 2 hours... if it was an easy fix, things would be online again already. My understanding is even basic building systems are still offline.

Originally Posted by digdugdog

Wow, glad I activated a partial backup... better than nothing! Good luck to those working on this!

Yes, we highly suggest anyone with something critical down to pop it up in another DC asap as there has not been any clear updates it's better to be safe than have things down for a larger time.

Originally Posted by hermetek

Psychz just posted an update in their portal that says "ETA: it might be down the entire day"

Happy Memorial Day!

back online now.

Originally Posted by Sam.Serverfreak

back online now.

Where? We are still down

Looks like Hivelocity is online:

https://status.hivelocity.net/785705204