Cloudflare’s Current Expansion Is Different from the Others

In January, Cloudflare, which helps companies make their web services run faster and be more secure – and which more recently started to use its global data center network to also provide cloud computing services – said it would expand the network in the US with three dozen new locations. Shortly thereafter, the company said it would add even more locations in the US – about the same amount as in the first announcement.

It would use Vapor IO and EdgeMicro for the first and second batch of locations announced, respectively. Cloudflare’s network already spans some 200 cities in more than 90 countries, but this round of expansion is different from any infrastructure deployment the company’s done before. Vapor IO and EdgeMicro are both startups building out a new kind of digital infrastructure: small data center sites designed specifically for edge computing. Each is going about it in its own way, but the result as far as their client Cloudflare is concerned is the same: more Cloudflare computing infrastructure closer to more internet users.

Both startups are starting with a few cities in the US. Vapor IO has its first edge data centers live in Chicago, Pittsburgh, Atlanta, and Dallas. It is building the next ones in 16 more cities, planning to bring them online this year, and expects to finish another 16 cities by the end of 2021. EdgeMicro is live in Austin, Raleigh, and Tampa, expects to launch in four more markets by the end of the third quarter, and may consider sites in Europe and South America in the future, but not within the next 12 months.

Deploying that many sites in one burst isn’t new for Cloudflare, which has over the years developed a process for this. The company has deployed 30 sites around the world within one month twice before. But its deployments to date have been in traditional colocation data centers and in ISP facilities. It normally selects locations striving for maximum latency reduction for a maximum number of people. Having compute infrastructure physically close to a lot of people plays a big role here.

“We are within 100 milliseconds of 95 percent of the world's population, or 99 percent if you look at internet users in the developed world,” Nitin Rao, head of global infrastructure at Cloudflare, told Data Center Knowledge in an interview.

That’s already better than accessing a hyperscale cloud data center (which could be in Ashburn, Virginia, for example, or outside of Portland, Oregon) from a city like New York or Seattle, which can be up to 200 milliseconds, Vapor IO chief marketing officer, Matt Trifiro, suggested. “Just by deploying their existing equipment footprint into our facilities, they can get lower latency in specific regions,” Trifiro said. “Cloudflare is very publicly, very aggressively moving to the edge. They’ve got competitors, but they’re out there first.”

As Rao put it, “when it comes to edge computing, closer is better,” but it’s not only about proximity to the end user; it’s also about network architecture. How efficiently is the local network that connects an access point to the nearest peering exchange designed? How many network hops does a packet have to make before it hits the exchange? Is there a regional exchange, or does traffic have to be backhauled to a big metro a few hundred miles away?

Different Site Selection and Network Strategies

The catalyst for edge computing – in this particular case – is the ability to connect from the edge of the access networks all the way to the centralized cloud, Trifiro told us. As far as low-latency workloads are concerned, the weakest link today is the “middle mile,” the section of the network between those access networks and the interconnection point for regional data centers.

This is why Vapor spends a lot of time on painstaking site selection and assembly of middle-mile fiber routes. Those are the company’s key strengths, Trifiro said. “We draw a ring around the population center of a city and overlay the assets we have access to, like land and fiber routes,” he explained. “Then we use another 150 criteria as risk-mitigation factors (like population location and the locations of cell towers and exchange points) and pick typically three sites in a city.” The sites are interconnected with multiple, redundant fiber routes, forming a distributed data center. “We also connect to the last-mile networks, the cable head ends, and the telco baseband unit, as well as the regional data centers, and we do that with multiple [fiber routes], typically dark fiber, that we assemble ourselves.”

There’s normally a lot of fiber in that middle mile within a metro, Trifiro said, but it’s owned by many different companies, and it’s difficult to assemble the optimal routes Vapor needs. So, the company uses commodity optical equipment and its own fiber connections to bring middle-mile transport price three orders of magnitude down from what typical providers charge, he claimed.

Vapor’s software-defined networking platform cross-connects any two networks that either terminate at or are connected within one of its pods, based on agreements between its customers like Cloudflare and network providers like Verizon, Comcast, or Charter. The result is sub-five-millisecond latency between any two Vapor locations in a city, Trifiro said.

While Vapor IO picks cities based on population density, EdgeMicro puts its colocation facilities for edge computing where customers ask for them. “We’ve been working with Cloudflare on how [to] deploy edge sites into tier-two and tier-three markets, where there’s a lot of eyeballs and a lot of traffic but no internet exchange,” Jason Bourg, EdgeMicro’s VP of strategy, told us.

Combination of the two gives Cloudflare the broadest geographical coverage.

Within a city, EdgeMicro chooses sites that are closest to fiber interconnects. “Where the core routers sit for the internet exchange, if there is one, or the internet partners, whether that’s the ISPs or the ILECs or the MSO and the cable network,” Bourg said. Unlike Vapor, the company doesn’t buy dark fiber and interconnect its own data centers, but it does broker network-connection partnerships for Cloudflare to avoid the network backhaul that’s usually required otherwise in secondary markets.

“All the internet traffic in Austin, Texas, whether that’s the enterprise ISP or the cable networks, the MSOs, it’s all backhauled to Dallas,” Bourg explained. It’s the same elsewhere: traffic goes through the seven or nine internet “drains” in major metro areas like Chicago, New York, New Jersey, Miami, Atlanta, and Los Angeles. That’s not good for low-latency streaming applications like Cloudflare Workers (its serverless cloud computing service) or its content delivery network.

“What we’ve worked so closely with Cloudflare on is how do we work with our network partners, the ISPs, the cable networks, on private peering localized traffic solutions so that when Cloudflare deploys in a tier-two market, that internet and data traffic is not being backhauled to the largest internet drain that's close by,” Bourg said. “Anybody can buy boxes and push them to the edge. That's not the value of the edge. The value of doing something at the edge is how the traffic stays localized in that market, so you can create a better end-user experience for these low-latency applications.”

In the past, Cloudflare has had to find local ISPs to peer with directly when there isn’t an internet exchange in the location they need and on occasion has shipped servers to deploy inside the ISP infrastructure. EdgeMicro’s relationships could speed up getting these lower-latency connections. “Cloudflare has always said, you guys understand the network better than anybody else,” Bourg said.

Cloudflare and EdgeMicro said they would disclose official network throughput later, but Bourg said the company saw promising initial roundtrip statistics of between 50 and 75 milliseconds. “We believe we can take that from 75 milliseconds down to sub 20, maybe even sub 10 milliseconds.”

Different Power Densities

Each remotely-managed Vapor site has between 150kW and 180kW of power capacity across high-density 35kW racks, which Trifiro said was comparable to power density in hyperscale cloud data centers. Meanwhile, high density hasn’t been a particular point of focus for EdgeMicro.

“We're not building super-dense 20kW racks for our client environments for the simple fact that our clients … don't need them,” Bourg said. “Our clients are saying 8kW to 10kW per rack is more than enough for us. The Cloudflare deployment is smaller.”

But Vapor’s pods are designed to accommodate multiple customers per rack, so in its case higher density means the ability to support more customers per site.

Bourg suggested that the edge shouldn’t be viewed as competition for hyperscale cloud regions, where core services would still run, but as a different kind of cloud, which suits specific needs. “In the tier-two markets it could be hyper-local compute, gaming, CDN, or serverless development platforms like Cloudflare’s.”

Rao said all the locations, Vapor and EdgeMicro, will run Cloudflare’s full set of services. “All 26 million internet properties that run on our network and all the features they turn on run on every server in every part of the world,” he said.

Cloudflare’s goal is widest possible distribution to improve the developer experience, Rao explained. “Gone are the days where you had to choose where to run your code, and you had to choose between running your code in Ashburn or Seattle, or Sao Paulo. Increasingly, the adoption of serverless, like Cloudflare Workers, has made it more likely that you can write code, and it runs in a number of locations… You get the efficiency that the developers are used to, but you also get the performance benefits of a distributed network.”