Thread: IOFlood/We Love Servers - Downtime / Move

Just got this in the mail about an hour ago..... At least they're giving us a month's notice

Hi Tom,

This email is to notify you of an upcoming scheduled maintenance on your server.

As you may know, we currently host most of our services at of our own datacenter in Tempe, AZ and a smaller number are collocated at PhoenixNAP. Due to upcoming contract changes, it is no longer feasible for us to keep these services at PhoenixNAP and we will be moving these services to our own datacenter in Tempe, AZ.

This move will not have any impact on the network providers or IP addresses used by your server, which will remain unchanged. We are scheduling a short interruption of service, and want to give you as much notice as possible.

This notice applies to the following server: [ip removed]

Maintenance is scheduled for: Thursday, January 16, 2025, and will begin sometime between 10 AM and 4 PM Arizona Time.

This maintenance is expected to cause downtime of approximately 120 minutes, but possibly less or more time may be necessary. This is because we need to shut down your server to relocate it to our primary datacenter, located a 20 minute drive from its current location.

We will perform a proper shutdown of the server by issuing a control-alt-delete on the console. For most operating systems this will tell the OS to properly shut down all programs to prepare for a reboot. If your OS requires a different process for shutting down, please provide us with the root password and a set of commands you would like us to run prior to shutting off your server, and we will do that instead.

Again, this maintenance for [ip removed] here is scheduled for: Thursday, January 16, 2025, and will begin sometime between 10 AM and 4 PM Arizona Time.

I know that this sort of thing can be disruptive to your service, and we would not take this action if it were not necessary to continue providing reliable and efficient service. Once the maintenance has been completed, our staff will reply on this ticket to let you know the work is done. If you have any difficulties with your server after that point, please reply and we will do our best to help.

Please let me know if you have any questions and I will do my best to assist.

Thanks,

NOT a fan of servers being shut down for 6 hours in the middle of the day (yes, I know they CLAIM 2 , but realistically, it'll be 6, we all know it), at all.

At least they scheduled it, but come on people.... DO BETTER. Pay people more to do this overnight, not during the middle of the day when you KNOW this will impact your customers more.

Hi Tom,

I'm sorry to hear this will be impacting you. A small number of customers still at our old datacenter, PhoenixNap, are being migrated to our company-owned datacenter due to egregious colocation price increases at the end of our current contract term at PhoenixNap.

I've emailed you with a different proposed plan for migration that may suit you better.

I would mention it would have likely been less than 2 hours based on previous experience, but we prefer not to over promise in case something doesn't go smoothly.

Hello,

Impact is not an issue , obviously these things happen and will need to be taken care of. I fully get that

The problem though, is that it’s not just me that will be affected by this timing . This is why it’s important to let people schedule this when it’s convenient for them . Keeping things down during business hours is just bad all around

I’ll take a look at the email and get back with you

Tom

Hi Tom,

Thanks for the clarification.

Of course we try to be flexible and shift things around if someone has a preference, but we can't know ahead of time who would prefer what times.

For moving this number of servers, we do have to make a schedule ahead of time rather than ask people what they'd prefer -- you'd be surprised how many people don't reply to the migration notice whatsoever, before during or after.

Many people want it to occur during regular hours so they can make sure it comes back up properly while they are working. Some prefer weekends or after hours.

Based on responses from the past, many people are indifferent as long as downtime is minimized and the server comes back online properly. Achieving that is best done when the most senior staff are on shift. In case a server doesn't come back on its own we want skilled staff available -- and not sleep deprived -- to be able to troubleshoot.

Sounds sweet to me. Escaping the grasps of future infinitely increasing commercial colocation costs. Better then increasing customer pricing on servers.

Hello and good luck Gabe.

I think the notice is far enough in advance, a full month, for a planned temporary move to another server during the downtime period if required. Need to have that capability anyway in case of hardware failure or other major issue that takes a server offline.

You're absolutely right, a month's notice is beautiful. Better than most here. The problem is that this was originally scheduled to be down during business hours, which, let's be honest, doesn't work for most.

Transitions like this should be done when it's going to impact the least amount of people, which is why you see professional companies (not saying Gabe isn't professional, or these people are not) doing migrations and server moves overnight, typically. Yes, the client is notified, but they don't NOTICE anything at all , because it's done while they're sleeping.

I wonder what type of maintenance takes 6 hour of down time? It has been my experience that maintenance for the most part is non disruptive. But when it comes to network and power upgrades there is a work around but not mere 6 hours as stated here.

Originally Posted by ITivan80

I wonder what type of maintenance takes 6 hour of down time? It has been my experience that maintenance for the most part is non disruptive. But when it comes to network and power upgrades there is a work around but not mere 6 hours as stated here.

We had announced anticipated 2 hours of downtime within a larger window when the work may occur, not 6 hours. It is for a server migration from one datacenter to another, so some downtime is unavoidable.

Tom / @whmcsguru,

Gabe and his crew know what they're doing. As he's described, it's best to have the correct crew be running on all cylinders when performing a move like this. As a provider, it's incredibly difficult to move servers when each client would prefer. Best thing you can do is to communicate in advance your preferences and back your data up! They might be able to give you direct notifications or check on specific services when moving.

It sounds like you were given one month notice, which is more than other providers I know of can do! I've known some to perform midnight moves without telling anyone they even moved the hardware. Unfortunately I've even known this to happen with colocation and not just dedicated hardware!

Best wishes to you and Merry Christmas, Sir!

While I do understand Tom's concern and dissatisfaction with the daytime migration, I also understand IOFLOOD's reason for the timing. I believe we all wish this could type of move could be accomplished with zero impact to every client, but we don't live in a perfect world. One month's advance notice is commendable.

Hey Tom,

Your WHT sign-up date shows the year 2002. That means you have been online for a fairly long time.

Have you ever thought about multi-homing some of the important stuff so you can survive real physical issues, such as server migration or facility fires?

Anyone here remember the Evocative DC fire in Secaucus last year? Or QTS before that?

Your initial response comes off as a little passive-aggressive (we all know it). It's almost as if this post was somewhat of rage bait for others, in addition to a place to voice your concerns.

You've been heard. Maybe you should move the servers home and self-host them at your primary residence to avoid future issues that are beyond your control.

Or, as someone put it once... DO BETTER.

Yes, I'm very well aware that they handled this (somewhat) well when it comes to notifications. The only issue is being forced into a daytime schedule. What they should have done was let clients know this was happening, give them a window (or multiple windows) and ask which works for them. That's proper customer service. Yes, I know, it involves a bit of an inconvenience for the people doing the work, but given that they're asking people to put their business on hold, with just 30 day's notification? Yeah that 'bit of an inconvenience' is the least that can be done. This is how it's done professionally.

No, I should not be obligated to fire up another server to cover what I'm already paying good $$$ for. That's even more $$$ out of my pocket in a market that's constantly shrinking budgets

Originally Posted by whmcsguru

Yes, I'm very well aware that they handled this (somewhat) well when it comes to notifications. The only issue is being forced into a daytime schedule. What they should have done was let clients know this was happening, give them a window (or multiple windows) and ask which works for them. That's proper customer service. Yes, I know, it involves a bit of an inconvenience for the people doing the work, but given that they're asking people to put their business on hold, with just 30 day's notification? Yeah that 'bit of an inconvenience' is the least that can be done. This is how it's done professionally.

No, I should not be obligated to fire up another server to cover what I'm already paying good $$$ for. That's even more $$$ out of my pocket in a market that's constantly shrinking budgets

I hear you Tom, I really do. Sometimes, the practical and pragmatic implications outweigh the disruptive nature of a move such as this (during the day). You're not wrong at all and there is a lot of value in what you are saying. I'm just trying to provide a different perspective. When we built out our DC footprint, we moved from an unfavorable landlord who played some pretty nasty games, and we did our moves at night. Our window was from 10PM - 4AM PST. Our clients spanned the globe, so it was daytime for someone. Our staff, after the nights we did the moves (which was Friday/Saturday/Sunday nights only), were completely spent. Due to move schedule(s) and moving over 500 physical machines (we hired a moving company to also assist with the moves) it was no trivial task and took months. At night, people are more tired, what would happen if there was a car accident, or a server sustained some physical damaged because XYZ reason (and they do happen). The truth is, I think more mistakes happen at night. We were VERY lucky and had no issues, but I can't tell you how many times I've heard of nightmare situations occurring at night.

So conceptually sure, a move at night is great, but if/when something happens, you're not dealing with the freshest staff, maybe a network config doesn't port over properly, or a server dies in transport. The shorter window then becomes the longer one.

Just my $.02.

-Steven

Originally Posted by insiderhosting

So conceptually sure, a move at night is great, but if/when something happens, you're not dealing with the freshest staff, maybe a network config doesn't port over properly, or a server dies in transport. The shorter window then becomes the longer one.

Been doing server moves for decades now, and no, overnight isn't always the best approach, which is why it's best to approach clients and say "hey, this is happening, you need to schedule a time to do so. If not, then this will be done at XXX time. Here's the available times we have". Pretty much ticks all the boxes . Yes, it means multiple schedules, which may not do best for the company, but it's more about keeping clients happy

Originally Posted by whmcsguru

Been doing server moves for decades now, and no, overnight isn't always the best approach, which is why it's best to approach clients and say "hey, this is happening, you need to schedule a time to do so. If not, then this will be done at XXX time. Here's the available times we have". Pretty much ticks all the boxes . Yes, it means multiple schedules, which may not do best for the company, but it's more about keeping clients happy

It sounds more like a 'move of servers' than a 'server move' to be a bit pedantic. Keeping it simple and during business hours makes it much more likely to be successful. Picking machines at random from racks to move them at different times while under time pressures with tired staff is asking for trouble. If you think some customers are unhappy with the time of the move just wait until things go wrong during the move and getting things fixed drags on for hours/days since so much time was wasted doing things inefficiently. Customers quickly forget how accommodating you were when things go wrong.

Myself I would not subject staff to your version - too much extra work and stress with the risk of things going south which can be very demoralizing. Doing a rack a time means small issues - like busted rack ears on one server in the middle - don't cause issues which they would if the server underneath needed to be moved hours earlier. I might offer to arrange a different move time for a few customers on a contact us if the time of the move would be a major problem basis. Would probably move those customers twice - once to another rack then to the new location so the bulk of server moves are full racks with one or two empty slots.

Originally Posted by whmcsguru

Yes, I'm very well aware that they handled this (somewhat) well when it comes to notifications. The only issue is being forced into a daytime schedule. What they should have done was let clients know this was happening, give them a window (or multiple windows) and ask which works for them. That's proper customer service.

You want them to reach out to each customer and get a time that works best for each customer? That is not practical as everyone would have a different preference and they'd just be going back and forth for days when it can be done seamlessly in less than 2 hours. Also, like they said, often customer's don't even respond to these type of emails so you want them contacting each multiple times to try to get an answer on what time works for them? Again, not practical and a big waste of time when they could be putting their full focus on moving all servers to their datacenter in a timely matter - which is what they're doing. I am not sure how this was handled "somewhat" well. To me, it looks like it's being handled very well. Communication is key and you got that with a month heads-up.

Originally Posted by SetraHost

You want them to reach out to each customer and get a time that works best for each customer? .

They already have to reach out to these customers to let them know this is happening. There is no need to reach out to customers individually at all, or to do anything more than simply add scheduled move times, telling the customer to schedule during these times if they want, or it will be moved during the quoted time.

Tom, it looks like we're just over two weeks away from the data center migration. Would love to hear how things went after the move. Again, Gabe and his crew do a great job managing things!

If you're on older hardware (especially hard drives), best make a backup of things.