Thread: Quadranet downtown LA

Originally Posted by coldthink

It seems that things are developing in a good direction. Next, it depends on when the service can be restored.

How do you know it's going in the right direction? I still don't see any response or any IP coming online.

Originally Posted by ITCompany

How do you know it's going in the right direction? I still don't see any response or any IP coming online.

Nothing has changed yet as far as I can tell. Their internal private IP network still works and that is probably how a previous poster is able to access their server from another server on the same internal network.

I can access my server on an IP6 address but that doesn't help me very much. I am currently configuring some of my back end stuff to use that instead of IP4 but it still doesn't address the elephant in the room.

Thanks for the tip. At least I'm able to access my server through IPV4 on the "Asian optimized subnet".

https://bgp.tools/prefix/66.212.30.0/23

Fortunately for us we only have one small subnet from them. From what I have been told, they lost control of IPs and they can't be recovered. A little investigating and its appears they lost all of 216.144.224.0/20.

I'd guess that management made a mistake giving back the wrong IPs to ARIN. Ever since Edge Centres took over QuadraNet I've seen some bigger customers leaving them. Maybe they have unused IP space they don't need or want to pay fees on anymore which led to this.

I wouldn't expect much communication on this. With all past outages and events I've never seen management get involved with communicating to customers what is happening until many days, even weeks after with a RFO. Even now with this issue their status page still says "We are currently experiencing a networking issue". Their helpdesk technicians only know that the IPs cannot be recovered and new ones need to be assigned. After this level of mistake though and them losing control of their website DNS for 3 days, I'd be worried this would happen again on the new IPs they are assigning. Hopefully they can get this resolved for all affected customers quickly.

Originally Posted by UnfinishedSentenc

Correction, my IP6 sub is working, so I can connect to my server via that now. Unfortunately, most of my stuff on this server is still using IP4

Yeah, I think ip6 is still working, as all the google mail (typically on ip6) seems to be getting delivered.

Very disappointing and unprofessional. One can only speculate but it appears the IP ranges were sold to HostPapa. During acquisitions often the acquiring firm desires the subscribers but not the IP address assignments that come with it, which are often sold off or in some cases simply leased. Assuming there was a plan to sell off excess IP assets, two weeks time to renumber out of the old IP assignments could have simply been offered.

Originally Posted by SolaDrive - John

Fortunately for us we only have one small subnet from them. From what I have been told, they lost control of IPs and they can't be recovered. A little investigating and its appears they lost all of 216.144.224.0/20.

I'd guess that management made a mistake giving back the wrong IPs to ARIN. Ever since Edge Centres took over QuadraNet I've seen some bigger customers leaving them. Maybe they have unused IP space they don't need or want to pay fees on anymore which led to this.

I wouldn't expect much communication on this. With all past outages and events I've never seen management get involved with communicating to customers what is happening until many days, even weeks after with a RFO. Even now with this issue their status page still says "We are currently experiencing a networking issue". Their helpdesk technicians only know that the IPs cannot be recovered and new ones need to be assigned. After this level of mistake though and them losing control of their website DNS for 3 days, I'd be worried this would happen again on the new IPs they are assigning. Hopefully they can get this resolved for all affected customers quickly.

I am on a different sub so it's not just the 216.x.x.x sub that is not working.

We cannot access internal network over VPN

Originally Posted by mr fixit

We cannot access internal network over VPN

Yes, their VPN is still not working either.

The status page shows down now and nothing else, ridiculous but glad to see I am not the only one with servers down

If they are going to sell their IPs why didn’t they give us time to renumber- even Verizon is more competent than this

I am seeing a new IP4 address range added in my Quadranet customer portal.

Originally Posted by UnfinishedSentenc

I am seeing a new IP4 address range added in my Quadranet customer portal.

I am looking on this page and still showing me our old IP range
https://manage.quadranet.com/client/..._ip_groups.php

do you have another link where you can see it?

Originally Posted by ITCompany

I am looking on this page and still showing me our old IP range
https://manage.quadranet.com/client/..._ip_groups.php

do you have another link where you can see it?

It may be different for you or they haven't added yours yet. For me, I go to Devices > View Devices then click on the relevant server for details. In there is an IP Assignments tab. I click on that.

In that link you gave I see it under "Default IP group"

Public Network IP Range --New 01/06/2025 x.x.x.x/28.

They haven't updated my ticket so it's probably not completely set up yet.

Complain here to the boss

https://www.linkedin.com/in/jonathan-eaves

If enough people demand answers we may get some

Here is some background / research I did on the IP space a user provided in this forum post.

• On December 14, 2015 QuadraNet acquired 216.144.224.0 - 216.144.239.255 (216.144.224.0/20) from ARIN and was using exclusively within their network.
• On December 13, 2024 QuadraNet no longer held control of the /20 and instead broke it up.
• On December 13, 2024 the IPv4 Space: 216.144.228.0 - 216.144.235.255 (216.144.228.0/22, 216.144.232.0/22) was acquired by ColoCrossing / HostPapa. QuadraNet still had control over 216.144.236.0/22
• On January 2nd, 2025 the IPv4 space: 216.144.236.0 - 216.144.237.255 (216.144.236.0/23) was acquired by ColoCrossing / HostPapa.
• The remaining IP space 216.144.224.0 - 216.144.227.255 (216.144.224.0/22) and 216.144.238.0/23 (216.144.238.0 - 216.144.239.255) are still in control by QuadraNet.

ColoCrossing / HostPapa was a long time PacificRack / QuadraNet customer and in Q3/Q4 of 2024 they moved out of QuadraNet's Downtown Los Angeles facility (530 West 6th Street). They are now in the West 7 center datacenter at 1200 West 7th in Los Angeles. The facility has other companies in their like LeaseWeb. The facility use to host Equinix (LA5) before they moved out. I believe 1200 West 7th use to have horrible power issues before they went underwent renovations in the last 10 years -- I am going off memory on the power issues so take that with a grain of salt.

We can only assume / guess that when ColoCrossing / HostPapa moved out from QuadraNet they bought IP space from QuadraNet -- 2,560 IPv4s. If we use IPv4.global / Hilco as a benchmark for the going market rate of IPv4 its around $34/IPv4 or roughly $87,040.

If you had servers directly from QuadraNet that was in the IP space acquired by ColoCrossing / HostPapa then you know why things are on longer working.


All this information was compiled by public and historical data from ARIN as well as my extensive knowledge of the datacenter market in Southern California.

This begs the question again why didn't they just renumber everyone before selling off the IPs. Why would they sell IPs? One would think not selling them is a way to keep your customer from moving.

This is great information, but my question is: how can they handle it so poorly?

Originally Posted by redrooster

This begs the question again why didn't they just renumber everyone before selling off the IPs. Why would they sell IPs? One would think not selling them is a way to keep your customer from moving.

Was the IPv4 QuadraNet giving you now owned by ColoCrossing / HostPapa? Or is the IP space still owned by QuadraNet?

Originally Posted by Gesundheit

Here is some background / research I did on the IP space a user provided in this forum post.

• On December 14, 2015 QuadraNet acquired 216.144.224.0 - 216.144.239.255 (216.144.224.0/20) from ARIN and was using exclusively within their network.
• On December 13, 2024 QuadraNet no longer held control of the /20 and instead broke it up.
• On December 13, 2024 the IPv4 Space: 216.144.228.0 - 216.144.235.255 (216.144.228.0/22, 216.144.232.0/22) was acquired by ColoCrossing / HostPapa. QuadraNet still had control over 216.144.236.0/22
• On January 2nd, 2025 the IPv4 space: 216.144.236.0 - 216.144.237.255 (216.144.236.0/23) was acquired by ColoCrossing / HostPapa.
• The remaining IP space 216.144.224.0 - 216.144.227.255 (216.144.224.0/22) and 216.144.238.0/23 (216.144.238.0 - 216.144.239.255) are still in control by QuadraNet.

ColoCrossing / HostPapa was a long time PacificRack / QuadraNet customer and in Q3/Q4 of 2024 they moved out of QuadraNet's Downtown Los Angeles facility (530 West 6th Street). They are now in the West 7 center datacenter at 1200 West 7th in Los Angeles. The facility has other companies in their like LeaseWeb. The facility use to host Equinix (LA5) before they moved out. I believe 1200 West 7th use to have horrible power issues before they went underwent renovations in the last 10 years -- I am going off memory on the power issues so take that with a grain of salt.

We can only assume / guess that when ColoCrossing / HostPapa moved out from QuadraNet they bought IP space from QuadraNet -- 2,560 IPv4s. If we use IPv4.global / Hilco as a benchmark for the going market rate of IPv4 its around $34/IPv4 or roughly $87,040.

If you had servers directly from QuadraNet that was in the IP space acquired by ColoCrossing / HostPapa then you know why things are on longer working.


All this information was compiled by public data and historical data from ARIN and my extensive knowledge of the datacenter market in Southern California.

Good summary of the likely cause but there are many other subnets in this too, since 67.215.224.0/19 is also affected. The following are all showing as hostpapa now ,

67.215.233.0/24 , 67.215.234.0/24 , 67.215.235.0/24 , 67.215.236.0/24 , 67.215.237.0/24 , 67.215.238.0/24 , 67.215.239.0/24 , 67.215.250.0/24 , 67.215.251.0/24 , 67.215.252.0/24 , 67.215.253.0/24 , 67.215.254.0/24 , 67.215.255.0/24

We used to have multiple /25's in a couple of those blocks but luckily moved most things due to constant power issues in 530 W 6th Street , so now only have a few servers remaining and they are affected by this. Got a response early on from quadranet that they lost control of the blocks and will need to get new IPs , but have not heard anything back since.

The reply from the staff sounded like this caught them by surprise so who knows what's going on over there.

My apologies, we are not a customer of theirs. Just weighing in my thoughts This calamity is quite the odd case.

What's more annoying is that as soon as there's a problem, they stop replying to tickets. If I hadn't come to WebHostingTalk, I wouldn't even have known there was a problem with their IP addresses.

Originally Posted by Scott.Mc

... but luckily moved most things due to constant power issues in 530 W 6th Street ...

The reply from the staff sounded like this caught them by surprise so who knows what's going on over there.

That has been our experience ever since EdgeCentre's acquired QuadraNet in early 2024. We were a colocation customer with our own racks in both of their Los Angeles locations (530 West 6th and 6171 West Century). As soon as things started slipping (all of their sales staff laid off, network and power issues, chiller failure, no more 24/7 on site support, management issues, etc.) we moved out.

We now have multiple racks at CoreSite in Los Angeles and experience has been night and day - power, network and service are all far better. Even our downstream customers of VM and bare metal services are happier. Lucky for us, we own our IP space.


However the real issue is not that customers have to re-ip their servers, it is the fact that their was no advanced notice when clearly Quadranet knew ahead of time this was going to happen but for what ever reason did not tell customers.


I hope the current QuadraNet leadership makes this right with customers. Gaining the IP space back is not possible at this point.

How is it even possible to make such a colossal error? What's even worse is their follow up. Twelve hours now and not a peep other that "uh, looks like we lost your IPs bro".

This is odd. I'm in 104.129.3.0/24, which is advertised as 104.129.0.0/18.

I did a Zenmap of 104.129.0.0/18, and most /24 nets all return something up and running in Quadranet. But nothing in my x.x.3.0/24. How could the little /24 I'm in be dead, while the broader /18 is alive?