Thread: On-site data center technicians - need to have or just on call?

I'm curious about everyone's thoughts/opinions about on-site technicians at data centers versus just having technicians on-call? The first data center I worked at years ago had technicians on-site 24/7/365, but later switched to having them on-call. As a client, I'd be concerned about response times should one or more of my servers needed technical assistance from my data center.

Originally Posted by SenseiSteve

I'm curious about everyone's thoughts/opinions about on-site technicians at data centers versus just having technicians on-call? The first data center I worked at years ago had technicians on-site 24/7/365, but later switched to having them on-call. As a client, I'd be concerned about response times should one or more of my servers needed technical assistance from my data center.

Don't underestimate the ability of a 24/7/365 staffed NOC to achieve worse response times than you'd expect from on-call : P

In general, I would rather it be staffed properly, especially during ordinary hours. It can be hard to get good quality staff to be willing to work overnight, and then twiddle their thumbs with nothing to do, no ongoing training, no coworkers to interact with, and no management oversight. So on that basis I wouldn't be too bothered by not having someone overnight just on the basis that, even if you did have someone there, you likely would be better off waiting until ordinary business hours to lodge your request anyway.

It's something of a red flag if nobody is there on a scheduled basis during regular business hours. That said, there's plenty of cases where you'd avoid using the DC remote hands and get a third party or direct employee to handle things for you, even if the DC is staffed 24/7, so in practical terms it may not always matter.

Originally Posted by funkywizard

Don't underestimate the ability of a 24/7/365 staffed NOC to achieve worse response times than you'd expect from on-call : P

In general, I would rather it be staffed properly, especially during ordinary hours. It can be hard to get good quality staff to be willing to work overnight, and then twiddle their thumbs with nothing to do, no ongoing training, no coworkers to interact with, and no management oversight. So on that basis I wouldn't be too bothered by not having someone overnight just on the basis that, even if you did have someone there, you likely would be better off waiting until ordinary business hours to lodge your request anyway.

It's something of a red flag if nobody is there on a scheduled basis during regular business hours. That said, there's plenty of cases where you'd avoid using the DC remote hands and get a third party or direct employee to handle things for you, even if the DC is staffed 24/7, so in practical terms it may not always matter.

When you say especially during ordinary hours, I wonder about their clients in other time zones, especially those across the globe 8, 10 and often 13 hours ahead of the data center's time zone. I understand there are situations where in practical terms it doesn't always matter.

Originally Posted by SenseiSteve

I'm curious about everyone's thoughts/opinions about on-site technicians at data centers versus just having technicians on-call? The first data center I worked at years ago had technicians on-site 24/7/365, but later switched to having them on-call. As a client, I'd be concerned about response times should one or more of my servers needed technical assistance from my data center.

It depends on what you're selling. Are you selling a proper datacenter with 24/7 staff or are you selling a lights out facility that "Bob" can go check on if he needs to?

I hear what you're saying, Aaron, but yes, I am talking about a proper data center (one of substantial size) where the lights are never turned off. A data center that hosts racks for other well known hosting providers and clients from around the globe.

Originally Posted by SenseiSteve

I hear what you're saying, Aaron, but yes, I am talking about a proper data center (one of substantial size) where the lights are never turned off. A data center that hosts racks for other well known hosting providers and clients from around the globe.

A substantially sized datacenter will be expected to have 24/7 staffing on site, if for no other reason than to ensure the security and uptime of the facility.

Originally Posted by SenseiSteve

I hear what you're saying, Aaron, but yes, I am talking about a proper data center (one of substantial size) where the lights are never turned off. A data center that hosts racks for other well known hosting providers and clients from around the globe.

Then yes. 24/7. You should be fully staffed as well, not a skeleton crew on off hours. If you're running a moderate sized network as well, most larger peers will require you to have a fully functional NOC 24/7.

Originally Posted by SenseiSteve

When you say especially during ordinary hours, I wonder about their clients in other time zones, especially those across the globe 8, 10 and often 13 hours ahead of the data center's time zone. I understand there are situations where in practical terms it doesn't always matter.

Certainly our customers from Asia are more likely to submit tickets later at night. But in general, the workload is much higher from 9a-5p than from midnight - 6am. Even rather busy operations will be down to just one guy with nothing to do during the overnight shift. So if you need anything that's not super basic or anything that requires approval or escalation, it won't be getting done until morning even if there is someone there to warm a chair.

Originally Posted by SenseiSteve

I'm curious about everyone's thoughts/opinions about on-site technicians at data centers versus just having technicians on-call? The first data center I worked at years ago had technicians on-site 24/7/365, but later switched to having them on-call. As a client, I'd be concerned about response times should one or more of my servers needed technical assistance from my data center.

Until that on-call staff found a better 8 hours job. You can't expect an on-call staff that he will be available anytime you want so achieving that 24/7 support is not possible in an on-staff call.

Some data centers have on-site staff that you can pay per hour if you need them though.

Some data centers have moved over to on-call instead of 24/7. Some of the data centers are doing on-call to help chore up their budgets

It depends on the labor work that needs to be done. IHM would have their DC techs on-call weekends and weekdays one person per shift.

Originally Posted by selinux

It depends on the labor work that needs to be done. IHM would have their DC techs on-call weekends and weekdays one person per shift.

We are a small, unmanaged, bare metal provider and we still have 3 technicians on site 24/7. One person per shift is unsafe. What if that person were to have a heart attack, or hurt themselves?

On call means "wait until it breaks." (Then hope they come in, investigate and address it soon. ) Prevention and Mitigation should be the priority.

Our primary data center is staffed 24/7/365, no matter the day or time. The daytime staff are more knowledgeable, but the night time crew is willing to work with you if you need to push buttons, read things, etc. I keep close relationships with facility and tech members, it makes things go smoothly!

Primary sites should be fully staffed IMHO, although I understand on-call IF you own your own DC and you know the needs.

24 hours/day all the way back to the 90s. I can't imagine the risks and failure modes that aren't addressed by not having anyone on site at one point or another.

On site staff ought to be the norm. It's not a luxury unless you have steady state boring big corporation client base. But those customers are going to catch you napping without staff quickly and it's going to be ugly (if they don't require such anyways).

I've been the guy going to the DC in the past to service customer gear. Spent many hours deaf and freezing in the racks over night with maybe 1-2 people present in the DC.

Contract on-call labor is fine for small businesses and lean startups.

You should have a plan, a threshold, where when so many customers, or revenue, that you staff properly.

Sure, I realize we have issues with labor idly sitting there 24/7/365. Labor's agreement is they sacrifice their time for facing the unknown. Some days labor has straight rotten day dealing with dumpster fire. And sure, probably many low flow days.

It always amazes me that so many business owners look at staffed DC as lazy people and inadequate. I think it happens, but often is the result of management hiring under performers or having a toxic workplace.

Do you pay your DC staff bonuses and hazard pay for days that are truly lousy? Probably not. So why think other times they are being lazy or anything. Can't win on both sides of the transaction.

FYI: I am long ago only a contractor. But I am observant of the industry friction with employees.

I feel this is highly dependent on the actual data center in question. Some data centers have remote hands help which are just NOC security guards who are useless. Others have much more sophisticated staff. Oddly the more expensive data centers are usually the ones with the useless staff and they charge double the second tier data center rates. If the second tier data center has superior help, then not only are you being charged less than a DRT for example, but they will get the work done for you in half the time and may even figure out your problem before you know what it is. It definitely does not hurt to keep your equipment at least in a place where there is good full service remote support, certainly if you tinker with gear a lot. Not only that, in general my experience has been that the colocation provider with good support are usually "run" better overall and geared towards helping smaller businesses in my experience. You may try to ask for 2 hours per year free remote hands in your contract. Many data centers will accept this concession and throw it in your agreement. They do not want to lose a deal over something which costs them literally nothing! That is my advice, so I hope that lets you choose the data center with good remote hands and worry less about the cost since you may get some free hands.

If you have IPMI, and a Lantronix Spider, the data centers may not charge you just to plug in your own KVM into a port or move it about. That only takes a few seconds and many larger DCs may do that for you without providing remote hands or charging for it. From what I recall DRT many years ago used to allow us to move a KVM port around once in a while or hit a reboot without having to pay for remote hands. Policy may be different now, but again you have leverage when starting a new contract, so ask for it.. FYI there are some data centers left that still provide free remote hands, both in LA and NYC. I would dig for those on Google.

I just want to add one thing again, what some smaller providers have done is hire an IT MSP on a retainer basis for break/fix standby. They will pay $X for 5-10 hour blocks of time, and when the poop hits the fan the IT team will dispatch down to the DC to do the work which has already been paid for. Then you simply make this on site IT support a line item in your budget. This may be cheaper than remote hands in some facilities too depending on where you live or where the equipment is.