Another Thing You Lose When You Move to the Cloud

Over the past two years or so, the infrastructure team behind Etsy – the internet store for all things vintage, handmade, or otherwise unique – has been moving the site from a colocation data center in New Jersey into Google’s cloud.

They’re doing it to support Etsy as it scales while taking advantage of Google’s top-notch machine learning infrastructure – but also its top-notch sustainability practices. Google claims that it operates some of the world’s most efficient data centers, and the only company that’s been able to truly compete with it for the title of the world’s biggest corporate buyer of renewable energy is Facebook.

By moving to the cloud, Etsy is gaining the ability to scale relatively easily and bring its portal to artists, craftspeople, and their customers anywhere around the world. It’s also gaining access to all the tools Google is constantly building for its cloud users and to all the latest hardware the giant is constantly integrating into its platform. But one of the things the ecommerce site is giving up in the move is a clear picture of the relationship between its software architecture and its energy use.

Using a colocation provider, that relationship has always been visible, Emily Sommer, a software engineer at Etsy, said while sitting on a panel at the recent DCD>San Francisco conference. With the site moving to the cloud, “we are losing some of that insight,” she said.

Few companies do this, but Sommer and her colleagues have made the energy impact of things users do on their site one of the key metrics to watch. They’ve tracked the way their servers’ energy consumption jumps when a Big Data job runs to show a visitor personalized search results, for example, and kept an eye on the overall number of kilowatt-hours consumed during each site visit.

Cloud providers don’t share with customers their individual energy use. Google and Microsoft may publish their total energy consumption numbers in annual sustainability reports (Amazon doesn’t disclose even that much), but they don’t break out consumption by individual location, let alone by individual server – and much less by individual virtual machine.

But sharing the data with customers could help make a dent in the numbers in those annual reports, Sommer suggested. “If you can give customers insight into their individual consumption, then we can make changes at the base level,” she said. “If all your consumers can save energy it can add up.”

She was addressing Gary Demasi, Google’s senior director for data center energy and location strategy, who was sitting on the same panel. It’s complicated, he replied, but instead of addressing her specific concern, he first shifted gears to one of the daily struggles of his own job: getting more utilities to be transparent about the mix of energy sources on their grids.

The puzzle he and his team have been trying to solve lately is matching Google’s energy consumption with renewables hour-for-hour. That means instead of buying enough renewable energy to match the company’s total consumption in a year (something Google already does), they want to make as much clean energy available on all the utility grids powering Google data centers or offices as those facilities consume at any given moment. Having more real-time data about grid-energy fuel mix would help solve that puzzle.

Giving Sommer the data she wants would be a big technical challenge, he said. A cloud availability zone usually consists of multiple data centers, and a user’s virtual machine or container could be on any of the thousands upon thousands of physical servers in one of those facilities. That would make measuring the exact amount of energy each user’s applications are consuming extremely complicated, Demasi said.

But, if he and his team eventually solve the puzzle of powering Google with renewables around the clock, Sommer’s concern will be a non-issue, he suggested: “If we move to a zero-carbon grid, we don’t have anything to worry about.”

Coming from Demasi, that’s fair enough. After all, his job is to power all of Google and to power it sustainably, not to build tools for cloud customers. It’s also easy to see why giving customers insight into their individual energy use would be hard. However, isn’t solving complex technological challenges what Google – and all of Silicon Valley – has always prided itself on?