cisco ucs sd card – AgileOps

Troubleshooting The vMotion migration failed because the ESX host were not able to connect over the vMotion network.

Ibrahim Quraishi — Mon, 10 Nov 2014 22:02:26 +0000

Hi All,

Today I will take you thought the troubleshooting for vMotion using CLI, If you have not done network troubleshooting on esx host you will find some really good esxi commands on this post. And if you are using a converged infrastructure its important to understand end to end connectivity especially if you have different vlans for each service like vMotion FT, Management traffic and VM-Traffic.

I build a brand new Data Center and after configuring the dv switch I decided to test the vMotion first and this is the error I saw on the first attempt the vm was stuck at 14 % and after a while it gave the below error message.

as shown below:

Migrate virtual machine error message

If you read the above error message you will find the IP’s of the host which I am migrating from and to.

for example lets consider the below two host this troubleshooting is done on esx-02. vmnic2 is FT vmnic3 is vMotion so I am comparing the mac address of booth the host on the ucs service profile to compare tht the vnic template has the right mac address and if they are connected to the right vlan.

esx-01

vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0d 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0c 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

esx-02

vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:19 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:18 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

check the mac address on ucs and on the esx host for the vmnic2 ( ft) vmnic3 (vmotion)
check the mtu it should be 9000 although this shuld not affect but I have changed them from 1500 mtu to 9000 mtu.
check the nics with mac address
esxcfg-nics -l
check the vmkernal nics
esxcfg-vmknic -l
ping from the vnic using
# vmkping -I vmk2 192.168.242.1 this is using vmk2 interface which is for vmotion
you will notice its failing initially because the uplink on the dv switch was first given to both the ft and vmotion nics
I then moved the dvuplink_1 to unused as I and it was my mistake which is why it was not working agian.
vmkping -I vmk2 192.168.242.1 the ping test failed again
then I moved the dvuplink_1 in active and moved dvuplink_2 to unused and the result is

The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools. Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ # esxcfg-nics
esxcfg-nics [nic]
-s|–speed Set the speed of this NIC to one of 10/100/1000/10000.
Requires a NIC parameter.
-d|–duplex Set the duplex of this NIC to one of ‘full’ or ‘half’.
Requires a NIC parameter.
-a|–auto Set speed and duplexity automatically. Requires a NIC parameter.
-l|–list Print the list of NICs and their settings.
-r|–restore Restore the nics configured speed/duplex settings (INTERNAL ONLY)
-h|–help Display this message.

~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:06:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:14 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1 0000:07:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:1a 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic10 0000:10:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:10 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic11 0000:11:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:11 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic12 0000:12:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic13 0000:13:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:10 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic14 0000:14:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:0c 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic15 0000:15:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:19 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:18 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic4 0000:0a:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:13 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic5 0000:0b:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:17 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic6 0000:0c:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:12 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic7 0000:0d:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:16 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic8 0000:0e:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:11 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic9 0000:0f:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:15 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 192.168.248.2 255.255.254.0 192.168.249.255 xx:xx:xx:a3:01:14 1500 65535 true STATIC
vmk1 1 IPv4 192.168.240.2 255.255.254.0 192.168.241.255 xx:xx:xx:68:7c:e3 9000 65535 true STATIC
vmk3 1 IPv4 192.168.237.2 255.255.255.0 192.168.237.255 xx:xx:xx:67:0d:34 1500 65535 true STATIC
vmk4 1 IPv4 192.168.245.2 255.255.255.0 192.168.245.255 xx:xx:xx:68:07:a0 1500 65535 true STATIC
vmk5 1 IPv4 192.168.244.2 255.255.255.0 192.168.244.255 xx:xx:xx:62:60:66 1500 65535 true STATIC
vmk6 129 IPv4 192.168.245.102 255.255.255.0 192.168.245.255 xx:xx:xx:64:9c:a8 1500 65535 true STATIC
vmk7 129 IPv4 192.168.244.102 255.255.255.0 192.168.244.255 xx:xx:xx:65:b6:41 1500 65535 true STATIC
vmk2 129 IPv4 192.168.242.2 255.255.254.0 192.168.243.255 xx:xx:xx:65:ff:87 9000 65535 true STATIC

~ # vmkping -I vmk2 192.168.242.1 – [ping to the esx-01 failed]
PING 192.168.242.1 (192.168.242.1): 56 data bytes

— 192.168.242.1 ping statistics —
3 packets transmitted, 0 packets received, 100% packet loss
~ # vmkping -I vmk2 192.168.242.2 – [ this is the same host I am troubleshooting so its ping is ok.]
PING 192.168.242.2 (192.168.242.2): 56 data bytes
64 bytes from 192.168.242.2: icmp_seq=0 ttl=64 time=0.081 ms
64 bytes from 192.168.242.2: icmp_seq=1 ttl=64 time=0.027 ms
64 bytes from 192.168.242.2: icmp_seq=2 ttl=64 time=0.055 ms

— 192.168.242.2 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.027/0.054/0.081 ms
~ #
~ # vmkping -I vmk1 192.168.242.2 [ vmk1 is the vmkernal port of FaultTolarant so it will not ping as its on a different subnet and vlan]
PING 192.168.242.2 (192.168.242.2): 56 data bytes

— 192.168.242.2 ping statistics —
2 packets transmitted, 0 packets received, 100% packet loss

~ # vmkping -I vmk1 192.168.240.2
PING 192.168.240.2 (192.168.240.2): 56 data bytes
64 bytes from 192.168.240.2: icmp_seq=0 ttl=64 time=0.084 ms

— 192.168.240.2 ping statistics —
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.039/0.062/0.084 ms
~ # vmkping -I vmk1 192.168.240.1
PING 192.168.240.1 (192.168.240.1): 56 data bytes

— 192.168.240.1 ping statistics —
2 packets transmitted, 0 packets received, 100% packet loss

Now it is clear that the MAC and NIC are presented correct from UCS service profile and we have verified the MAC address and pinging from the vmkernal ports proofs that its is still not working.

I have check that they vnic template is connected to the right vlan on ucs. So next thing I have done is check the dvport group and I found that I had two uplinks on active which was wrong as one uplink is only for FT which has vlan 20and the other uplink is for vMotion which is on 21 vlan.

So, I have moved the dvuplink2 to unused as its for FT. on the dvport group which is for vMotion as shown below:

— 192.168.242.2 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.040/0.061/0.088 ms
~ # vmkping -I vmk2 192.168.242.1
PING 192.168.242.1 (192.168.242.1): 56 data bytes
64 bytes from 192.168.242.1: icmp_seq=0 ttl=64 time=0.428 ms
64 bytes from 192.168.242.1: icmp_seq=1 ttl=64 time=0.197 ms
64 bytes from 192.168.242.1: icmp_seq=2 ttl=64 time=0.219 ms

— 192.168.242.1 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.197/0.281/0.428 ms

UCS Blade unable to ping or connect to vCenter

Ibrahim Quraishi — Fri, 31 Oct 2014 22:30:11 +0000

Hi All, I came across a strange issue where 2 blades was unable to ping I could get 2 ping and then Request time outs Problem : 2 Servers( in the same chassis server 3,4 ) unable to ping their gateway. Ping drops are after 1-2 packets. We cannot ping/ssh to them from outside and they cannot be added to vCenter. Other blades in the same chassis are working ( 1,2) Steps Taken :

Confirmed that we do not have any IP conflicts
Made sure that we do not use the same subnet anywhere else in the network
Also checked and confirmed that the Mgmt(CIMC) subnet is different

Started a ping from inside a Host ( Server-3 ) to check if you are able to get to the gateway
Checked vobd logs to find if you can find any entrie for duplicate ip/mac
Checked the Mgmt Interface : vmk0
It is connected to Port Group : Management Network on VLAN ID, Check if we are using the right vlan
Found that the MAC Address of the vmk0 is same as vnic (vmnic0) : This is a known issue VMware KB article here. This is the vnic mac address of the service profile which is attached to the esx blade.
Deleted and re-created the vmk0 interface
Confirmed that the MAC Address has changed
Started a continuous ping which work fine now
Host can now be added to the vCenter

So let me go through of the process of how we found the duplicate mac address. The vnic mac address was taken by vmknic as you might know vmknic mad starts with 00:50:56 but in my case it was same as the mac address of vnic from the service profile. login to the host and give the command #esxcfg-nics -l

this gives the list of vnics assigned to the service profile, note the management nics in my case vnic 0 and 1 then give the below command to get the vmkernal nics #esxcfg-vmknic -l

as you can see above the mac address is not starting from 00:50:56 and it is been duplicated from the vnic template from the service profile vnic0 which is the root cause for the connectivity issue. Now to fix this we have to delete the virtual switch and re-create it. #esxcfg-vswitch -l This lists the virtual switches

#esxcfg-vmknic -d -p ‘Management Network’ you have to specify exact name of the port Now we will need to add the vmknic with the same Management network #esxcfg-vmknic -a -i -n ‘Management Network’ and also give the command #esxcfg-vswitch -l to check the Management network created as shown below. note you can not give a gateway at this point you can specify frm DCA

you should now be able to ping to the host as it has got a new mac address to check it you can give the command #esxcfg-vmknic -l

for a continus ping you can give #ping -c 100 this will give 100 continuous pings Reference: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031111]]>

Cisco UCS Firmware Overview

Ibrahim Quraishi — Wed, 03 Sep 2014 20:50:01 +0000

Cisco UCS FlexFlash Configuration of Secure Digital (SD) Card So, for the newbee to Cisco UCS, its confusing about the firmware. In this post I am trying to clarify few aspects of this, feel free to post comment below if you have any questions and I will get back to you. The CISCO UCS firmware is bundled into two packages as given below: 1. Infrastructure Firmware In Infrastructure Software firmware we update the firmware of the following

Cisco UCS Manager software
Kernel and system firmware for the fabric interconnects
I/O module firmware

2. Server Firmware Cisco UCS B-Series Blade Server Software Bundle

CIMC firmware
BIOS firmware
Adapter firmware
Board controller firmware
Third-party firmware images required by the new server

This also includes the firmware of Adapter, BIOS, FC Adapters, HBA Option ROM, Storage, Controller, Local Disk Cisco UCS C-Series Rack-Mount UCS-Managed Server Software

CIMC firmware
BIOS firmware
Adapter firmware
Board controller firmware
Third-party firmware images required by the new server

As most of you might be aware the UCS manager is the GUI Java based which allows you to fully manage the CISCO Unified Computing System. Note I have noticed that the Java version 7 update 40 is only compatible with the UCS Manager. So, where does the UCS Manager run ? This was one of my first questions, Well the Software of the UCS Manager runs on the FI ( Fabric Interconnect 62xx) as we have a pair of Fabric Interconnect it has got resilience. The UCS System is designed to be highly scalable and available, so upgrading the firmware should not bring any outage to the environment. However Cisco still suggests to arrange a Change windows and get this done out of hours or the quite period of your business. This UCS Manager is bundled in the Infrastructure Firmware Software along with the Kernal update of the FI which is the NXOS, Finally we have the I/O Module firmware which is a part of the Infrastructure firmware software too. The Other Bundle is the Server Firmware which has the firmware for BIOS, the Blades, Adapter, Board controller etc. If you want to learn more about the process of doing a Firmware update please checkout my link CISCO UCS Firmware Update Process Hope this information is useful, feel free to comment.]]>

Cisco UCS FlexFlash Configuration of Secure Digital (SD) Card

Ibrahim Quraishi — Thu, 21 Aug 2014 19:32:54 +0000

Cisco UCS FlexFlash /SDCard Configuration for Booting Server Operating System. As most of the Cisco UCS users might be aware that Cisco has now add a lot of improvements and new features on the latest upgrade 2.2(1e). One of the feature which we have started using is Cisco FlexFlash which is an SD Card. Its fully support if you buy the SD Card from Cisco and we can even do mirroring on the Flash cards for redundancy, we will be installing ESXi Server 5.5 on these blades after mirroring them. So where and How to install the Secure Digital (SD) Card ? If the CISCO UCS B200 Blade is in the Chassis you will need to power it off and slide it out just half way through and you should find the SD Card slot on the left for more information on how about this can be found on the following link https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/B200M3.html As you might be aware that CISCO UCS blade is just as equal to a brick and all the configuration is done on the logical server called Service profile. you will need to create a new / clone an existing Service profile template and modify with the below changes which I have shown below on this post. You will have to create a boot policy with CD Drive (legacy ) and SD Card and Some time you may end up getting this error ‘ advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall ‘ if you see it follow my blog post. I have to make you aware that if you want to use SD card with Mirroring or RAID 1 you have to create the following new policies and create a Boot policy as mentioned above. 1. Disk policy with Mirror and any configuration 2. We have to create two Scrub policy under policy ( you can call it any thing you want which makes you understand what configuration you have selected)

Flex Flash Scrub
No Scrub

So where will you find the Flex Flash Controller? Click on the Left pan on Equipment and Select the Chassis then the Server, on the right you will find the below details click on Inventory and select the last tab which is Storage on the Storage tab you will find FlexFlash Controller Note: this FlexFlash is only support if you are running 2.2 (d1) firmware. Once you assign the new service profile policy, we will see the below error on the FlexFlaash Cards

Go to the service Profile and select the Policies and change the Scrub policy
in this create 2 policy the default is no scrub but you can create one called FlexFlash Scrub and the other one No Scrub

To create the scrub policy click on Service profile of the server and click Policy > Scrub and click Create Scrub Policy

You will have to use the first policy for Flex flash Scrub which should have the below settings.

FlexFlash Scrub: should set to Yes

You then need to create a new policy for No-Scrub as shown below :

After creating two policy this is the tricky bit to have the SD card as Mirror you will need to do two reboots first set the scrub policy to FlexFlash-Scrub ( note this will scrub the SD cards) and click Save Do the following steps:

Go to the server and Click server Maintenance this will scrub the flash
Reboot the server in the following manor, go the the Equipment tab select the Chassis and the server Click on ‘Server Maintenance’ and click on ‘Re-acknowledge’ server

Acknowledge for Reboot You will find the rediscovery Once its configured go back to service profile Go to > Polices Select Scrub Policy and change it to No-Scrub from the drop down menue which should look like below: 5. Go to the server and Click server Maintenance this will Mirror and pair the SD card to Raid1/ Mirroring 6. Reboot the server in the following manor, go the the Equipment tab select the Chassis and the server Click on ‘Server Maintenance’ and click on ‘Re-acknowledge’ server Acknowledge for Reboot After the server does the rediscovery and boots successfully, You will see that this is successfully paired with is RAID 1 Mirroring. Please note the Raid Status : Enabled Paired Error message if the Raid is not working, this means you have to go through the process again. always make sure you only change the scrub policy on the specific server profile and not the service profile template as this will create problems on the working profiles. Another example of error is Raid Status Disabled How to test ? The only confirmation we can get is on the below image where it says Raid Status Enabled Paired, however I have tested swapping the SD cards out one by one and the ESXi Server boots successfully you will immediately see the error message that the pair is disabled. Unfortunately you will need to pull the server out to change swap the SD cards but hay at least you will find out if the mirroring is working. Also you will need to remember that after installing ESXi on SD card will give you an error message on the ESXi host that ‘The ESXi host does not have persistent storage’ you will need to add the storage and a scrach partition. To create a scrach partition follow the below VMware KB. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033696 o~~r I might create a blog on how to do it later…~~ My friend Vikram has just posted the process on his blog www.viktec.com System logs on host are stored on non-persistent storage Please find the new updates the latest firmware is bringing. Release 2.2(1b) adds support for the following: IPv6 Management Support Cisco Integrated Management Controller (CIMC) In-band Management Fabric scaling: VLAN, VIFs, IGMP, Network Adapter Endpoints Uni-Directional Link Detection (UDLD) Support User Space NIC (usNIC) for Low Latency Support for Virtual Machine Queue (VMQ) C-Series Servers Direct Connect to FI without FEX Two-factor Authentication for UCS Manager Logins VM-FEX for Hyper-V Management with Microsoft SCVMM Direct KVM Access Server Firmware Auto Sync Enhanced Local Storage Management Flash Adapters and HDD Firmware Management Precision Boot Order Control Secure Boot UEFI Boot Support FlexFlash (Local SD card) Support Trusted Platform Module (TPM) Inventory DIMM Blacklisting and Correctable Error Reporting C-Series Board Controller Firmware Management Additional resources related to this post: If you want to learn more about the process of doing a Firmware update please checkout my link CISCO UCS Firmware Update Process If you want to learn more about the process of doing a Firmware update please checkout my link advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall