UCS Blade unable to ping or connect to vCenter

Hi All, I came across a strange issue where 2 blades was unable to ping I could get 2 ping and then Request time outs Problem : 2 Servers( in the same chassis server 3,4 ) unable to ping their gateway. Ping drops are after 1-2 packets. We cannot ping/ssh to them from outside and they cannot be added to vCenter. Other blades in the same chassis are working ( 1,2) Steps Taken :

1. Confirmed that we do not have any IP conflicts
2. Made sure that we do not use the same subnet anywhere else in the network
3. Also checked and confirmed that the Mgmt(CIMC) subnet is different

1. Started a ping from inside a Host ( Server-3 ) to check if you are able to get to the gateway
2. Checked vobd logs  to find if you can find any  entrie for duplicate ip/mac
3. Checked the Mgmt Interface : vmk0
4. It is connected to Port Group : Management Network  on VLAN ID, Check if we are  using the right vlan
5. Found that the MAC Address of the vmk0 is same as vnic (vmnic0) : This is a known issue VMware KB article here. This is the vnic mac address of the service profile which is attached to the esx blade.
6. Deleted and re-created the vmk0 interface
7. Confirmed that the MAC Address has changed
8. Started a continuous ping which work fine now
9. Host can now be added to the vCenter

    So let me go through of the process of how we found the duplicate mac address. The vnic mac address was taken by vmknic  as you might know vmknic mad starts with 00:50:56 but in my case it was same as the mac address of vnic from the service profile. login to the host and give the command #esxcfg-nics -l

this gives the list of vnics assigned to the service profile, note the management nics in my case vnic 0 and 1 then give the below command to get the vmkernal nics #esxcfg-vmknic -l

as you can see above the mac address is not starting from 00:50:56 and it is been duplicated from the vnic template from the service profile vnic0 which is the root cause for the connectivity issue. Now to fix this we have to delete the virtual switch and re-create it. #esxcfg-vswitch -l This lists the virtual switches

#esxcfg-vmknic -d -p ‘Management Network’ you have to specify exact name of the port Now we will need to add the vmknic with the same Management network #esxcfg-vmknic -a -i <IPaddress> -n <Netmask> ‘Management Network’  and also give the command #esxcfg-vswitch -l  to check the Management network created as shown below. note you can not give a gateway at this point you can specify frm DCA

 you should now be able to ping to the host as it has got a new mac address to check it you can give the command #esxcfg-vmknic -l

  for a continus ping you can give #ping <gateway ip> -c 100 this will give 100 continuous pings   Reference: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031111]]>