Finding – and Fixing – Hidden Kubernetes Costs

If you've ever deployed containerized applications at scale, you know just how valuable an orchestration solution like Kubernetes is: It automates deployment and management tasks that would otherwise be impossible to perform manually across a large environment. From a cost perspective, however, a heavy-duty orchestrator is not always the ideal solution. There are signs that cost pressures are encouraging some organizations to look beyond Kubernetes, or at least seek ways of controlling the various Kubernetes costs.

What Does Kubernetes Cost?

If you use Kubernetes, you add a variety of costs to your total IT bill.

Kubernetes Costs: Distribution

One of the costs of Kubernetes software comes with the distribution you use. Unless you use generic open source Kubernetes, which is rare because it requires so much effort, you will need a Kubernetes distribution. Most Kubernetes distributions cost money, so that's one expense to factor into your bill.

Kubernetes Costs: Hosting

Another cost is the infrastructure that you have to devote to running Kubernetes itself. The various Kubernetes services that run on master and worker nodes consume resources, and those resources cost money.

To be sure, the infrastructure consumed by Kubernetes itself is likely not huge compared to the resources that it takes to run your actual workloads. But it's not nothing. Kubernetes requires at least 300 megabytes of memory per node, for example, partly to run Kubernetes services and partly to support containers and pods. If your cluster has 100 nodes, the memory costs add up.

Kubernetes Costs: Management

A third obvious cost associated with Kubernetes is the work required to manage it. When you use Kubernetes, Kubernetes becomes another tool you have to learn and administer. Ideally, the effort required to use Kubernetes is outweighed by the work Kubernetes saves through orchestration, but the Kubernetes-management effort exists nonetheless.

Kubernetes Costs: Decrease in Visibility

A final cost-related challenge associated with Kubernetes stems from the fact that Kubernetes may make it more difficult to identify and address situations where you are under-utilizing hardware resources. If you are running workloads in generic cloud services, like virtual machine instances, you can use a variety of tools and strategies (like autoscaling and tiered storage) to avoid unnecessary costs. Many of these methods don't work with Kubernetes, however.

In addition, Kubernetes in general doesn't care whether you have more nodes running than you need, or you have provisioned more storage volumes than necessary, or you have assigned more resources via quotas than you should have. It will happily orchestrate your deployments even if they have twice as many resources assigned to them as they need.

Nor does Kubernetes offer any built-in tooling (beyond the ability to access very basic monitoring data, at least) to help you figure out whether your clusters and deployments are over-provisioned. You would have to figure that out yourself by tediously mapping resource allocations on the underlying hardware to workloads within your Kubernetes environment.

In other words, then, Kubernetes adds a layer of obfuscation that makes it more difficult to optimize costs than it would be if you didn't have Kubernetes standing in between your infrastructure and your applications.

Addressing Kubernetes Costs

Faced with these added costs of Kubernetes, developers and IT teams may react in a variety of ways.

One is to forego automated container orchestration altogether. That's not a practical solution in most cases, however. Again, it's basically impossible to orchestrate containers at scale if you do it manually.

Another approach is to use a lighter-weight orchestration solution, like Docker Swarm. With Swarm, you'll still face the cost-visibility issues described above, but your management and infrastructure costs are likely to be lower, because Swarm is a simpler orchestrator.

Along similar lines, there are orchestrators like HashiCorp Nomad, which – in a sign that the market is beginning to react to the cost challenges of container orchestration – recently added dynamic application sizing and recommendations that may help reduce overspending on infrastructure. Kate Lehman, director of corporate communications at HashiCorp, told me in an email that the company added the feature because "we regularly hear from organizations that they are unable to proactively manage cloud spend and often see increased costs from unused resources, e.g., memory or CPU utilization."

Some teams might also opt for Kubernetes distributions that support cluster autoscaling, a feature that automatically adds or removes nodes from your cluster based on workload demand. Most of the public cloud Kubernetes-as-a-service solutions, like Google Kubernetes Engine and Elastic Kubernetes Service, do this. You can also use cluster autoscaling with standalone Kubernetes distributions like Rancher, at least under certain configurations. The major downside of cluster autosizing is that it only works when you are running Kubernetes in a public cloud.

Conclusion

Kubernetes is a great tool – indeed, it has almost become a must-have tool for any organization that deploys containers – but cost-management is not its strong suit. If you run Kubernetes, you face a variety of added costs that wouldn't apply, or would be lower, when using other deployment technologies.

There are several ways to address this issue. You could forego Kubernetes altogether in favor of a lower-cost orchestrator, or one that bundles infrastructure-optimization features into the tool. Kubernetes distributions that support cluster autoscaling can help reduce costs, too, provided they work with your environment.

The bottom line: To avoid shooting yourself in the foot by spending more on Kubernetes than the tool saves you, make sure you factor cost management into your container orchestration strategy.