Hi All,

Today I will take you thought the troubleshooting for vMotion using CLI, If you have not done network troubleshooting on esx host you will find some really good esxi commands on this post. And if you are using a converged infrastructure its important to understand end to end connectivity especially if you have different vlans for each service like vMotion FT, Management traffic and VM-Traffic.

I build a brand new Data Center and after configuring the dv switch I decided to test the vMotion first and this is the error I saw on the first attempt the vm was stuck at 14 % and after a while it gave the below error message.

as shown below:

Migrate virtual machine error message

If you read the above error message you will find the IP’s of the host which I am migrating from and to.

for example lets consider the below two host this troubleshooting is done on esx-02. vmnic2 is FT vmnic3 is vMotion so I am comparing the mac address of booth the host on the ucs service profile to compare tht the vnic template has the right mac address and if they are connected to the right vlan.

esx-01

vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0d 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0c 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

 esx-02

vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:19 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:18 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

1. check the mac address on ucs and on the esx host for the vmnic2 ( ft) vmnic3 (vmotion)
2. check the mtu it should be 9000 although this shuld not affect but I have changed them from 1500 mtu to 9000 mtu.
3.  check the nics with mac address
   esxcfg-nics -l
4.  check the vmkernal nics
   esxcfg-vmknic -l
5.  ping from the vnic using
   # vmkping -I vmk2 192.168.242.1 this is using vmk2 interface which is for vmotion
   you will notice its failing initially because the uplink on the dv switch was first given to both the ft and vmotion nics
   I then moved the dvuplink_1 to unused as I and it was my mistake which is why it was not working agian.
6.  vmkping -I vmk2 192.168.242.1 the ping test failed again
   then I moved the dvuplink_1 in active and moved dvuplink_2 to unused and the result is

The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools. Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ # esxcfg-nics
esxcfg-nics <options> [nic]
-s|–speed <speed> Set the speed of this NIC to one of 10/100/1000/10000.
Requires a NIC parameter.
-d|–duplex <duplex> Set the duplex of this NIC to one of ‘full’ or ‘half’.
Requires a NIC parameter.
-a|–auto Set speed and duplexity automatically. Requires a NIC parameter.
-l|–list Print the list of NICs and their settings.
-r|–restore Restore the nics configured speed/duplex settings (INTERNAL ONLY)
-h|–help Display this message.

~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:06:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:14 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1 0000:07:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:1a 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic10 0000:10:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:10 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic11 0000:11:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:11 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic12 0000:12:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic13 0000:13:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:10 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic14 0000:14:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:0c 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic15 0000:15:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:19 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:18 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic4 0000:0a:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:13 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic5 0000:0b:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:17 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic6 0000:0c:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:12 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic7 0000:0d:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:16 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic8 0000:0e:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:11 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic9 0000:0f:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:15 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 192.168.248.2 255.255.254.0 192.168.249.255 xx:xx:xx:a3:01:14 1500 65535 true STATIC
vmk1 1 IPv4 192.168.240.2 255.255.254.0 192.168.241.255 xx:xx:xx:68:7c:e3 9000 65535 true STATIC
vmk3 1 IPv4 192.168.237.2 255.255.255.0 192.168.237.255 xx:xx:xx:67:0d:34 1500 65535 true STATIC
vmk4 1 IPv4 192.168.245.2 255.255.255.0 192.168.245.255 xx:xx:xx:68:07:a0 1500 65535 true STATIC
vmk5 1 IPv4 192.168.244.2 255.255.255.0 192.168.244.255 xx:xx:xx:62:60:66 1500 65535 true STATIC
vmk6 129 IPv4 192.168.245.102 255.255.255.0 192.168.245.255 xx:xx:xx:64:9c:a8 1500 65535 true STATIC
vmk7 129 IPv4 192.168.244.102 255.255.255.0 192.168.244.255 xx:xx:xx:65:b6:41 1500 65535 true STATIC
vmk2 129 IPv4 192.168.242.2 255.255.254.0 192.168.243.255 xx:xx:xx:65:ff:87 9000 65535 true STATIC

~ # vmkping -I vmk2 192.168.242.1 –  [ping to the esx-01 failed]
PING 192.168.242.1 (192.168.242.1): 56 data bytes

— 192.168.242.1 ping statistics —
3 packets transmitted, 0 packets received, 100% packet loss
~ # vmkping -I vmk2 192.168.242.2   – [ this is the same host I am troubleshooting so its ping is ok.]
PING 192.168.242.2 (192.168.242.2): 56 data bytes
64 bytes from 192.168.242.2: icmp_seq=0 ttl=64 time=0.081 ms
64 bytes from 192.168.242.2: icmp_seq=1 ttl=64 time=0.027 ms
64 bytes from 192.168.242.2: icmp_seq=2 ttl=64 time=0.055 ms

— 192.168.242.2 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.027/0.054/0.081 ms
~ #
~ # vmkping -I vmk1 192.168.242.2 [  vmk1 is the vmkernal port of FaultTolarant so it will not ping as its on a different subnet and vlan]
PING 192.168.242.2 (192.168.242.2): 56 data bytes

— 192.168.242.2 ping statistics —
2 packets transmitted, 0 packets received, 100% packet loss

~ # vmkping -I vmk1 192.168.240.2
PING 192.168.240.2 (192.168.240.2): 56 data bytes
64 bytes from 192.168.240.2: icmp_seq=0 ttl=64 time=0.084 ms

— 192.168.240.2 ping statistics —
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.039/0.062/0.084 ms
~ # vmkping -I vmk1 192.168.240.1
PING 192.168.240.1 (192.168.240.1): 56 data bytes

— 192.168.240.1 ping statistics —
2 packets transmitted, 0 packets received, 100% packet loss

Now it is clear that the MAC and NIC are presented correct from UCS service profile and we have verified the MAC address and pinging from the vmkernal ports proofs that its is still not working.

I have check that they vnic template is connected to the right vlan on ucs. So next thing I have done is check the dvport group and I found that I had two uplinks on active which was wrong as one uplink is only for FT which has vlan 20and the other uplink is for vMotion which is on 21 vlan.

So, I have moved the dvuplink2 to unused as its for FT. on the dvport group which is for vMotion as shown below:

— 192.168.242.2 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.040/0.061/0.088 ms
~ # vmkping -I vmk2 192.168.242.1
PING 192.168.242.1 (192.168.242.1): 56 data bytes
64 bytes from 192.168.242.1: icmp_seq=0 ttl=64 time=0.428 ms
64 bytes from 192.168.242.1: icmp_seq=1 ttl=64 time=0.197 ms
64 bytes from 192.168.242.1: icmp_seq=2 ttl=64 time=0.219 ms

— 192.168.242.1 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.197/0.281/0.428 ms