Uncategorized – AgileOps

Announcing General Availability of vSphere 7…

Ibrahim Quraishi — Wed, 07 Oct 2020 08:46:02 +0000

Announcing General Availability of vSphere 7 Update Introduces several key enhancements that will allow you to deliver developer-ready infrastructure, scale without compromise, and simplify operations. vSphere with Tanzu, the fastest way to get started with Kubernetes workloads.

Announcing General Availability of vSphere 7…

On September 15 we announced the upcoming release of vSphere 7 Update 1. Today, I am excited to share that it is now generally available. If you missed our announcement webcast on September 15, no problem. You can watch a replay of the webcast “Up-Level to Developer-Ready Infrastructure” anytime you’d like. What’s new in vSphere 7 Update 1? vSphere 7 Update 1 introduces The post Announcing General Availability of vSphere 7 Update 1 appeared first on VMware vSphere Blog.

VMware Social Media Advocacy

VMworld 2020 and Discord sessions Day 1

Ibrahim Quraishi — Tue, 29 Sep 2020 11:27:27 +0000

VMware 2020 is here and I am very excited about it 3 days of back to back learning and innovation. I hope you all have registered for the sessions and have scheduled them.

I will be streaming some of my favourite sessions on DISCORD I am the leader of Vision and Innovation track on their check out HTTP://Discord

Agenda for today is the following

We had VMUG virtual party yesterday and it was good to have some a different virtual experience. I know not a lot of people had different views about this virtual event, anyway I think it was a good initiative.

https://www.vmug.com/events2/vmworld-2020

Hope you enjoy VMworld 2020!

Last chance to register for VMworld 2020!

Ibrahim Quraishi — Tue, 22 Sep 2020 17:31:25 +0000

Last chance to register for VMworld 2020!

Join us online for VMworld 2020. Learn about industry hot topics, preview new hands-on labs, and participate in networking events. Mark your calendar.

VMware Social Media Advocacy

Whats new in vSphere 7 U1

Ibrahim Quraishi — Mon, 21 Sep 2020 16:00:23 +0000

Whats new in vSphere 7 U1

Whats new in vSphere 7 U1 and VCF 4 Introducing the vSphere with Tanzu Grid the fastest way to provision the Morden application. Deliver Developer ready Infrastructure. vSphere 7 EVC on GPU vSphere 7 U1 max host per cluster increase to 96 ESXi Host Max VM’s per cluster 10,000 Max VM CPU 768 […]

VMware Social Media Advocacy

Cisco Live 2016

Ibrahim Quraishi — Sat, 13 Feb 2016 21:53:41 +0000

Hi All, It has been a long time since I have published any post. I have been quite busy these days. I am going to attend Cisco Live in Berlin, Germany next week. So I just thought this is going to be a good opportunity to share my views of Cisco Live and the latest and greatest technologies Cisco has to offer. For those of you who are interested to keep an eye on Cisco Live Cisco Live webcasts and broadcasts are free to attend! To attend a webcast, you will need a Cisco Live account. Register now for your free account or log in if you already have one The list of events are given below: https://www.ciscolive.com/emea/ ( Online Events – Cisco Live ) Stay tuned for more updates… ]]>

Troubleshooting The vMotion migration failed because the ESX host were not able to connect over the vMotion network.

Ibrahim Quraishi — Mon, 10 Nov 2014 22:02:26 +0000

Hi All,

Today I will take you thought the troubleshooting for vMotion using CLI, If you have not done network troubleshooting on esx host you will find some really good esxi commands on this post. And if you are using a converged infrastructure its important to understand end to end connectivity especially if you have different vlans for each service like vMotion FT, Management traffic and VM-Traffic.

I build a brand new Data Center and after configuring the dv switch I decided to test the vMotion first and this is the error I saw on the first attempt the vm was stuck at 14 % and after a while it gave the below error message.

as shown below:

Migrate virtual machine error message

If you read the above error message you will find the IP’s of the host which I am migrating from and to.

for example lets consider the below two host this troubleshooting is done on esx-02. vmnic2 is FT vmnic3 is vMotion so I am comparing the mac address of booth the host on the ucs service profile to compare tht the vnic template has the right mac address and if they are connected to the right vlan.

esx-01

vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0d 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0c 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

esx-02

vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:19 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:18 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

check the mac address on ucs and on the esx host for the vmnic2 ( ft) vmnic3 (vmotion)
check the mtu it should be 9000 although this shuld not affect but I have changed them from 1500 mtu to 9000 mtu.
check the nics with mac address
esxcfg-nics -l
check the vmkernal nics
esxcfg-vmknic -l
ping from the vnic using
# vmkping -I vmk2 192.168.242.1 this is using vmk2 interface which is for vmotion
you will notice its failing initially because the uplink on the dv switch was first given to both the ft and vmotion nics
I then moved the dvuplink_1 to unused as I and it was my mistake which is why it was not working agian.
vmkping -I vmk2 192.168.242.1 the ping test failed again
then I moved the dvuplink_1 in active and moved dvuplink_2 to unused and the result is

The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools. Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ # esxcfg-nics
esxcfg-nics [nic]
-s|–speed Set the speed of this NIC to one of 10/100/1000/10000.
Requires a NIC parameter.
-d|–duplex Set the duplex of this NIC to one of ‘full’ or ‘half’.
Requires a NIC parameter.
-a|–auto Set speed and duplexity automatically. Requires a NIC parameter.
-l|–list Print the list of NICs and their settings.
-r|–restore Restore the nics configured speed/duplex settings (INTERNAL ONLY)
-h|–help Display this message.

~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:06:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:14 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1 0000:07:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:1a 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic10 0000:10:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:10 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic11 0000:11:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:11 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic12 0000:12:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic13 0000:13:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:10 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic14 0000:14:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:0c 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic15 0000:15:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic2 0000:08:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:19 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:09:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:18 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic4 0000:0a:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:13 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic5 0000:0b:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:17 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic6 0000:0c:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:12 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic7 0000:0d:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:16 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic8 0000:0e:00.00 enic Up 20000Mbps Full xx:xx:xx:a3:01:11 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic9 0000:0f:00.00 enic Up 20000Mbps Full xx:xx:xx:b3:01:15 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 192.168.248.2 255.255.254.0 192.168.249.255 xx:xx:xx:a3:01:14 1500 65535 true STATIC
vmk1 1 IPv4 192.168.240.2 255.255.254.0 192.168.241.255 xx:xx:xx:68:7c:e3 9000 65535 true STATIC
vmk3 1 IPv4 192.168.237.2 255.255.255.0 192.168.237.255 xx:xx:xx:67:0d:34 1500 65535 true STATIC
vmk4 1 IPv4 192.168.245.2 255.255.255.0 192.168.245.255 xx:xx:xx:68:07:a0 1500 65535 true STATIC
vmk5 1 IPv4 192.168.244.2 255.255.255.0 192.168.244.255 xx:xx:xx:62:60:66 1500 65535 true STATIC
vmk6 129 IPv4 192.168.245.102 255.255.255.0 192.168.245.255 xx:xx:xx:64:9c:a8 1500 65535 true STATIC
vmk7 129 IPv4 192.168.244.102 255.255.255.0 192.168.244.255 xx:xx:xx:65:b6:41 1500 65535 true STATIC
vmk2 129 IPv4 192.168.242.2 255.255.254.0 192.168.243.255 xx:xx:xx:65:ff:87 9000 65535 true STATIC

~ # vmkping -I vmk2 192.168.242.1 – [ping to the esx-01 failed]
PING 192.168.242.1 (192.168.242.1): 56 data bytes

— 192.168.242.1 ping statistics —
3 packets transmitted, 0 packets received, 100% packet loss
~ # vmkping -I vmk2 192.168.242.2 – [ this is the same host I am troubleshooting so its ping is ok.]
PING 192.168.242.2 (192.168.242.2): 56 data bytes
64 bytes from 192.168.242.2: icmp_seq=0 ttl=64 time=0.081 ms
64 bytes from 192.168.242.2: icmp_seq=1 ttl=64 time=0.027 ms
64 bytes from 192.168.242.2: icmp_seq=2 ttl=64 time=0.055 ms

— 192.168.242.2 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.027/0.054/0.081 ms
~ #
~ # vmkping -I vmk1 192.168.242.2 [ vmk1 is the vmkernal port of FaultTolarant so it will not ping as its on a different subnet and vlan]
PING 192.168.242.2 (192.168.242.2): 56 data bytes

— 192.168.242.2 ping statistics —
2 packets transmitted, 0 packets received, 100% packet loss

~ # vmkping -I vmk1 192.168.240.2
PING 192.168.240.2 (192.168.240.2): 56 data bytes
64 bytes from 192.168.240.2: icmp_seq=0 ttl=64 time=0.084 ms

— 192.168.240.2 ping statistics —
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.039/0.062/0.084 ms
~ # vmkping -I vmk1 192.168.240.1
PING 192.168.240.1 (192.168.240.1): 56 data bytes

— 192.168.240.1 ping statistics —
2 packets transmitted, 0 packets received, 100% packet loss

Now it is clear that the MAC and NIC are presented correct from UCS service profile and we have verified the MAC address and pinging from the vmkernal ports proofs that its is still not working.

I have check that they vnic template is connected to the right vlan on ucs. So next thing I have done is check the dvport group and I found that I had two uplinks on active which was wrong as one uplink is only for FT which has vlan 20and the other uplink is for vMotion which is on 21 vlan.

So, I have moved the dvuplink2 to unused as its for FT. on the dvport group which is for vMotion as shown below:

— 192.168.242.2 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.040/0.061/0.088 ms
~ # vmkping -I vmk2 192.168.242.1
PING 192.168.242.1 (192.168.242.1): 56 data bytes
64 bytes from 192.168.242.1: icmp_seq=0 ttl=64 time=0.428 ms
64 bytes from 192.168.242.1: icmp_seq=1 ttl=64 time=0.197 ms
64 bytes from 192.168.242.1: icmp_seq=2 ttl=64 time=0.219 ms

— 192.168.242.1 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.197/0.281/0.428 ms

UCS Blade unable to ping or connect to vCenter

Ibrahim Quraishi — Fri, 31 Oct 2014 22:30:11 +0000

Hi All, I came across a strange issue where 2 blades was unable to ping I could get 2 ping and then Request time outs Problem : 2 Servers( in the same chassis server 3,4 ) unable to ping their gateway. Ping drops are after 1-2 packets. We cannot ping/ssh to them from outside and they cannot be added to vCenter. Other blades in the same chassis are working ( 1,2) Steps Taken :

Confirmed that we do not have any IP conflicts
Made sure that we do not use the same subnet anywhere else in the network
Also checked and confirmed that the Mgmt(CIMC) subnet is different

Started a ping from inside a Host ( Server-3 ) to check if you are able to get to the gateway
Checked vobd logs to find if you can find any entrie for duplicate ip/mac
Checked the Mgmt Interface : vmk0
It is connected to Port Group : Management Network on VLAN ID, Check if we are using the right vlan
Found that the MAC Address of the vmk0 is same as vnic (vmnic0) : This is a known issue VMware KB article here. This is the vnic mac address of the service profile which is attached to the esx blade.
Deleted and re-created the vmk0 interface
Confirmed that the MAC Address has changed
Started a continuous ping which work fine now
Host can now be added to the vCenter

So let me go through of the process of how we found the duplicate mac address. The vnic mac address was taken by vmknic as you might know vmknic mad starts with 00:50:56 but in my case it was same as the mac address of vnic from the service profile. login to the host and give the command #esxcfg-nics -l

this gives the list of vnics assigned to the service profile, note the management nics in my case vnic 0 and 1 then give the below command to get the vmkernal nics #esxcfg-vmknic -l

as you can see above the mac address is not starting from 00:50:56 and it is been duplicated from the vnic template from the service profile vnic0 which is the root cause for the connectivity issue. Now to fix this we have to delete the virtual switch and re-create it. #esxcfg-vswitch -l This lists the virtual switches

#esxcfg-vmknic -d -p ‘Management Network’ you have to specify exact name of the port Now we will need to add the vmknic with the same Management network #esxcfg-vmknic -a -i -n ‘Management Network’ and also give the command #esxcfg-vswitch -l to check the Management network created as shown below. note you can not give a gateway at this point you can specify frm DCA

you should now be able to ping to the host as it has got a new mac address to check it you can give the command #esxcfg-vmknic -l

for a continus ping you can give #ping -c 100 this will give 100 continuous pings Reference: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031111]]>

The system detected an address conflict for IP address 0.0.0.0 with the system after a reboot

Ibrahim Quraishi — Fri, 31 Oct 2014 15:13:22 +0000

Hi All, This week I have seen an interesting issue on Windows Servers 2008 /2012 . Just to let you know these are VM’s and I was unable to connect to these server. So I login on vCenter to check the windows host and found the server was having network error. After checking the event viewer i found this error The system detected an address conflict for IP address 0.0.0.0 with the system having network hardware address xx-xx-xx-CE-44-3F. Network operations on this system may be disrupted as a result. Time stamp 28/10/2014 hh:mm:ss. Quick Fix Reboot the server and it got the network back. I have to mention giving ipconfig was showing the correct IP address even when it was having the network error. Further investigation reviled few interesting and worrying facts, apparently this issue is only affecting Windows Vista and above, we also found that it is a known issue, However it will arise only when a windows server is rebooted. Root Cause The root cause is part of the detection flows defined by the RFC 5227 (IPv4 Address Conflict Detection). The error is caused by the method used by Windows to detect an address conflict (https://tools.ietf.org/html/rfc5227#section-2.1.1) and one of the packets used by the cisco security feature called “ip device tracking”, used for the NAC Layer 2 validation. Unfortunately the IOS version (15.2) used by the most of the Cisco core switches, enables this feature by default and there is no way to disable it. The only options are: downgrade the IOS or tuning some parameters. Solution a. downgrade the IOS b. modify a parameter of Cisco ‘IP device tracking’ feature in order to potentially solve this issue. On each interfaces ip device tracking maximum 0 References: httpss://social.technet.microsoft.com/Forums/en-US/82e7b9a6-c58e-4b73-9fac-8e73f3347b91/0000-ip-address-conflicts-on-the-network-vms-lose-network-after-reboot?forum=winserverhyperv https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/8021x/116529-problemsolution-product-00.html Other blog: https://networkengineering.stackexchange.com/questions/6617/cisco-3560x-switch-mac-reported-with-same-ip-as-other-equipment]]>

Cisco UCS Invicta Storage – How to Provision a LUN to ESX Host

Ibrahim Quraishi — Mon, 27 Oct 2014 17:00:17 +0000

Note:

Create a default LUN and add MAP ID 0 ‘zero’ when ever you are provisioning a new ESX rule of thumb for Invicta is add a default lun lun0_ default with MAP ID 0
Keep all the LUN’s with the same MAP id on each host when provisioning them. ( Caution : If you do not give the same MAP ID for a LUN which is already provisioned and the lun has VM running. The LUN comes up as a new lun to be added to VMware so the Administrator if he is not aware of this issue may end up formating it assuming its a new LUN to provision using VMFS 5.
You need to remember that you can right click and drag and drop luns to map them.

In our environment I have created a default lun with 10 GB and never provision this on VMware, however always add this default lun first with MAP ID 0 to any ESX host you want to provision. Login to the invicta with admin or superuser account.

Step 1: Creating a LUN

Click on LUN Configuration and LUNs and Click on Create LUN

As I have mentioned in the Notes at the top that we need to first create a default lun which is what I am creating below. just fill in the details I call it lun0_default

The size of this lun can even be 1 GB as you are not going to use this lun . I have put 10 GB a I have 65 TB of thin provisioned disk and I am not going to provision this lun its just what is recommended by Cisco documentation to have a default lun 0 provisioned on each host with MAP ID 0 I have created another LUN of 1 TB calling it lun1 as shown below

Step 2. Adding a Host Initiator Group

Click on LUN Configuration and Initiator Groups and Click on Create New Group

Give the server name or if you have 10 esx host to provision the full storage you can call the group and add all the iqn numbers there. – click on Create Group

After the group is created you will find it as shown below, you can right click and see the options you can perform. We need to Add Initiator to this group. In this case I am going to add only one esx host. Copy the iqn number of from the esx host and add it.

To get the IQN if you are using the iSCSI initiator you can go to the host Click on Configuration tab and click on Storage Aadapters and right click and create iscsi adapter. Copy the IQN number.

You will also need to add the iSCSI target ( Invicta iSCSI target) in the Dynamic Discovery tab

Once you get the IQN number copy the IQN number and past it in the WWPN/IQN field below and then you should be ready to provision LUNS.

It will look like this

Step 3: Provisioning LUNS to the ESX Host

To map a LUN right click the esx-01 Initiator Group you have just added and select the second option Map/Unmap Luns

This will take you to a next screen where you will see Avalable LUNs to be added at the bottom and provisioned luns on the top as shown below:

As I have mentioned in the Note that we can drag and drop luns you can select lun0_default and prop it on the top space to provision it. At this point it will automatically assign the lun0_defualt lun the MAP ID 0 . Click Create

Always add the default lun in my example lun0_default first to every esx host or Initiator Group and give it MAP ID 0, remember this is recommended by Cisco. After adding this you can go ahead and add any number of map id however remember to keep the MAP id unique for each lun to avoid problems with lun coming up as unformatted. So this is how you will find after provisioning both the luns keep in mind to have the same lun number and map id number to keep it simple as shown below:

]]>

advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall

Ibrahim Quraishi — Fri, 19 Sep 2014 09:59:21 +0000

Failure Reason: advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall, You might have come across this error. Although the above error looks like a very generic error when you apply a service profile to a Server. Most of the on-line search takes you in the direction of making sure you are using the legacy CD ROM drive. With the process of elimination I found that while using the Flex Flash ‘SD Card’ in Boot Option this error was trigged. once I removed the SD Card, I was able to apply the profile with out any problems. But I wanted to use the Flex Flash with RAID-1,( SD Cards) to boot. more info on Cisco UCS FlexFlash Configuration of Secure Digital (SD) Card As you might be aware that boot from SD card is a new feature introduced in version 2.2 , we then started looking at he version of UCS Manager and Firmware running on the environment. I had recently upgraded this UCS to the 2.2(1d) so that was not an issue. We thought to double check the firmware configuration, when going through this options we create a Host Firmware Policy specifically only for M3 and just selecting the M 200 M3 CIMC and Bios with the latest version of firmware. more info on firmware can be found CISCO UCS Firmware Update Process Once this option was applied to the Service profile The blade started to boot up. The only thing different to this was I had created a custom firmware on Host Firmware Policy called 2.2 firmware but ended up selected quite a lot of servers with latest version of software .. However in my UCS Environment I have only got M 200 M3. So to fix this issue, make sure in ‘Host Firmware Package’ make sure you have customized firmware profile for every different server you have, so that you can individually apply it to the service profile you are assign to that specific server. Example M2 and M3 Blades we can have two Host Firmware Packages which you can add to specific Service profiles individually. Additional resources: If you want to learn more about the process of doing a Firmware update please checkout my link CISCO UCS Firmware Update Process If you want to learn more about the process of doing a Firmware update please checkout my link Cisco UCS FlexFlash Configuration of Secure Digital (SD) Card ]]>