Why NIC Team Bandwidth Seems Limited

Q: I have configured a NIC team and am performing Live Migration over it, but I'm only seeing 1Gbps bandwidth used for the migration--why?

A: NIC Teaming uses several different algorithms to distribute traffic over the available network adapters. One of these modes is Hyper-V Port, which effectively distributes the virtual NICs of the virtual machines (VMs) over the available network adapters. This is a good option if you have more VMs than network adapters in the team. Never use this mode if the team is used for Live Migration--it won't distribute the traffic at all.

The other mode in Windows Server 2012 is Address Hash, which uses a combination of source and target IP address plus source and target ports (4-tuple hash, although other modes can be selected by using Windows PowerShell and setting the LoadBalancingAlgorithm parameter--but this won't help with the current problem).

The issue with this, when performing one or even multiple concurrent Live Migrations between two hosts, is the IP address and port information between the two servers are the same for all the Live Migrations between the two hosts (as the source and target IP would be the same and Live Migration always uses port 6600). Therefore they would use only one of the NICs in the team (see graphic below). This is because the distribution algorithm is based on IP and port, which are the same for all the Live Migrations between the two hosts.

Only if performing multiple concurrent Live Migrations to different target Hyper-V hosts would multiple network adapters in the source Hyper-V host team be used. For example, if the source Hyper-V host with the team was performing three concurrent Live Migrations, each Live Migration to a different host. This would allow multiple network adapters in the team to be used.

Basically, anytime you have a great deal of the same type of traffic going between the same two hosts, then NIC Teaming won't distribute the data between multiple network adapters because it has to be sure it's keeping the order of the packets. There has to be a difference in the source/destination ip/port for the traffic to distribute among the network adapters in the NIC team.