Eddie's Blog

Troubleshoot vMotion Error 195887167

Updated on 07/13/2016. See the update this post, I might find the ultimate solution, even I am still not sure what the cause of the issue.

Recently I had an issue to vMotion some VMs between the vSphere v 6.x cluster hosts. Long story short, here are the symptoms:

I consistently got the error “Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout” when vMotioning (host only, no storage vMotion) on some VMs, particularly the vCenter Server Appliance VM. I had two vCSA VMs. Both were have the same issue. But vCSA is not the only VM that I got the error.
I successfully vMotion some VMs between the hosts. The vMotion network configuration should be okay.
In other words, some VMs are okay; some are not. The size (CPU, RAM, storage) of the VMs does not seem the problem. The successfully vMotioned VMs can have more/less CPU, RAM, storage than the failed VMs.
The failed VM has more disks than other VMs. The vCSA VM created 11 disks by default.
When the VMs were powered off, vMotion successfully.
Restarted the hosts and restarted the VMs. No difference.
Verified no IP address conflicts.
Tried one VMkernel adapter for both management and vMotion or a dedicated VMkernel adapter for vMotion. No difference.
Tested vmkping successfully between the hosts.
In the vmkernel.log of the hosts, the error is “2016-05-18T22:47:34.959Z cpu15:39089)WARNING: Migrate: 270: 1463611379229538 D: Failed: Connection closed by remote host, possibly due to timeout (0xbad003f) @0x418000e149ee” on the destination host or “2016-05-19T18:29:23.930Z cpu1:130133)WARNING: Migrate: 270: 1463682486286991 S: Failed: Migration determined a failure by the VMX (0xbad0092) @0x41803a7f6993” on the source host.

Possible solutions

Remove the snapshot on the VM if it has one
- After removing the snap shot on one of vCSA VMs, vMotion worked fine. But another vCSA had no snapshot, it still failed.
Try using the vSphere Client instead of the vSphere Web Client. This worked on some VMs, but not always.
Assign VM’s network adapter to different port group; and change back to its original port group
- This seems the ultimate fix. After doing this, the vCSA VMs, which failed vMotion consistently, are vMotioned successfully.

Conclusion

I’m not sure the root cause of this issue. But it may relate to the network setting on the vSwitch or port group. Some hits about this: vMotion fails with the general system error: 0xbad003f (KB2008394)
These KBs are not the solution in my case:

07/13/2016 update

I run into the same error when migrating some VMs, including the vCenter 6.x appliance (vCSA), between hosts in the Vmware cluster. I am able to migrate other VMs. This leads me to believe the problem on the VM, instead of the VM infrastructure.
Within the VMs having this error, some of them, which I can power off, are migrated successfully. However, I can not power down the vCSA VM. Because I cannot perform the migration without the vCenter available.
I try assigning the NIC of the vCSA VM to different port group, and change back. However, I cannot do that, because this VM is the vCenter server and is configured with a distributed switch. If I change the vCSA to different port group (configured with a different VLAN), the vCenter server will be down (because its NIC is assigned to the wrong port group with the wrong VLAN); and I cannot change it back to the original port group with the right VLAN.
I try connecting directly to the ESXi host of the vCSA VM via the vSphere C# client, then assign the NIC of the vCSA VM to different port group. However, I cannot do that either. Because the host is only configured with the distributed switches, there is no other port group in the selection at this situation.

I try create a new port group with the ephemeral port binding on the distributed switch. This new ephemeral port group is available in the vSphere C# client when connecting to the host directly. Then I assign the VM to the ephemeral port group and change back. However, the migration still fails with the same error.
Since I could fix this issue last time by changing the port group and changing back, I guess that somehow reset the NIC on the VM or the virtual switch port to which the VM is connected. That gives me an idea to manually assign the VM to another virtual switch port.

Solutions

In the screenshot above, the NIC of the vCSA is assigned to port 214 on the vSwitch.
Log back in the vCenter via the vSphere C# client or Web client (I cannot see the ports on the distributed switch, nor change the port assigned to the VM’s NIC when connecting to the ESXi host via the vSphere C# client)
I find an used port on the same port group (e.g. port 423 in my case).

Edit the vCSA VM setting and assign its NIC to the unused port.
Then I can successfully migrate the vCSA VM to another host.

Conclusion

Changing the NIC to an used port will be my first attempt when this issue happens again (I bet it will happen).
I still don’t know the cause of the this issue.

10/15/2019 update

I got the exact error again when Storage vMotion a VM across two vCenter (from vCSA 6.0 to vCSA 6.5). Deleted the VM snapshot, reran the vMotion, and completed successfully.

Change Windows Server 2008 or 2012 Network Profile

Sometimes a Windows server is assigned to the incorrect network profile. It can cause applying the wrong Windows Firewall rules. Here is how to change its network profile.

For standalone server

Can change the profile to public or private; but cannot set to domain
For Windows Server 2012

Open PowerShell as administrator
Get-NetconnectionProfile | Set-NetconnectionProfile -NetworkCategory [Private | Public]

For Windows Server 2008 or 2012

gpedit.msc, Computer Configuration, Windows Settings, Security Settings, Network List Manager Profiles
Select the network name, Properties, Network Location
Under Location Type, select Private or Public

For domain joined server

Cannot change the profile manually
It’s determined by Network Location Awareness (NLA) service (see “Network Location Awareness (NLA) and how it relates to Windows Firewall Profiles”)
If the network profile is incorrect,

verify the server can contact a DC via UDP port 389
restart the Network Location Awareness service on the server

Extend Microsoft Cluster Shared Disk in VMware

A VM shared disk on Microsoft Cluster Service (MSCS) is running out of disk space. The VMs are on a single host (aka cluster in a box - CIB). I can think of two ways to expand the disk storage.

create a new big shared disk for the cluster, migrate the data, then change the new disk to the same drive letter as the original disk
extend the size of the existing shared disk

Obviously the latter seems simpler, but it requires special attention. The shared disk format in MSCS VMs must be in eager zeroed thick format. However, when extending an eagerzeroedthick VMDK, the extended chuck is in lazy zeroed thick format by default (reference “Extending an EagerZeroedThick Disk”. In my test, vSphere 6 has the same behavior)

Here is how I extend the MSCS shared disk

Power off both servers in the cluster
Increase the VMDK disk size. There are two ways:

GUI: edit the VM settings, increase the shared disk size
CLI: use vmkfstools -X <newsize> -d eagerzeroedthick <vmdkfile>

For more info about vmkfstools, see my vmkfstools Examples post

Using the GUI, the extended chuck will be in lazy zero thick format. The VM will fail to power on with the error “VMware ESX cannot open the virtual disk for clustering…”

There are two ways to convert the extended chuck to eagerzeroedthick format

Migrate the VM to another storage, and specify the eager zero thick format for the disk
Use vmkfstools -k <vmdkfile>

Once the entire shared disk is the eager zeroed thick format, the VM will be able to power on.
Extend the Windows partition as KB304736

vmkfstools Examples

When searching an issue on expanding a shared disk on Microsoft clustering VMs (CIB), I have learned more about the vmkfstools command.

The vmkfstools --help displays many options, but lack of explanation. So I document them here. (reference: vSphere Storage, Using vmkfstools)

# vmkfstools --help

OPTIONS FOR FILE SYSTEMS:

vmkfstools -C --createfs [vmfs3|vmfs5]
               -b --blocksize #[mMkK]
               -S --setfsname fsName
           -Z --spanfs span-partition
           -G --growfs grown-partition
   deviceName

           -P --queryfs -h --humanreadable
           -T --upgradevmfs
   vmfsPath
           -y --reclaimBlocks vmfsPath [--reclaimBlocksUnit #blocks]

OPTIONS FOR VIRTUAL DISKS:

vmkfstools -c --createvirtualdisk #[gGmMkK]
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               ]
               -a --adaptertype [buslogic|lsilogic|ide
                                |lsisas|pvscsi]
               -W --objecttype [file|vsan]
               --policyFile <fileName>
           -w --writezeros
           -j --inflatedisk
           -k --eagerzero
           -K --punchzero
           -U --deletevirtualdisk
           -E --renamevirtualdisk srcDisk
           -i --clonevirtualdisk srcDisk
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               |rdm:<device>|rdmp:<device>
                               |2gbsparse]
               -W --object [file|vsan]
               --policyFile <fileName>
               -N --avoidnativeclone
           -X --extendvirtualdisk #[gGmMkK]
               [-d --diskformat eagerzeroedthick]
           -M --migratevirtualdisk
           -r --createrdm /vmfs/devices/disks/...
           -q --queryrdm
           -z --createrdmpassthru /vmfs/devices/disks/...
           -v --verbose #
           -g --geometry
           -x --fix [check|repair]
           -e --chainConsistent
           -Q --objecttype name/value pair
           --uniqueblocks childDisk
   vmfsPath

OPTIONS FOR DEVICES:

vmkfstools -H --help

vmkfstools Command Syntax

vmkfstools options target

Options: separate into three types - File System Options, Virtual Disk Options, and Storage Device Options.
Target: partition, device, or path

File System Options

Listing Attributes of a VMFS Volume
The listed attributes include the file system label, if any, the number of extents comprising the specified VMFS volume, the UUID, and a listing of the device names where each extent resides.
vmkfstools -P -h <vmfsVolumePath>
vmkfstools -P -h /vmfs/volumes/netapp_sata_nfs1/
Creating a VMFS Datastore
vmkfstools -C vmfs5 -b <blocksize> -S <datastoreName> <partitionName>
vmkfstools -C vmfs5 -b 1m -S my_vmfs /vmfs/devices/disks/naa.ID:1
Extending an Existing VMFS Volume
vmkfstools -Z <span_partition> <head_partition>
vmkfstools -Z /vmfs/devices/disks/naa.disk_ID_2:1 /vmfs/devices/disks/naa.disk_ID_1:1
Caution: When you run this option, you lose all data that previously existed on the SCSI device you specified in span_partition.
Growing an Existing Extent
vmkfstools –G device device
vmkfstools --growfs /vmfs/devices/disks/disk_ID:1 /vmfs/devices/disks/disk_ID:1

Virtual Disk Options

Creating a Virtual Disk
vmkfstools -c <size> -d <diskformat> <vmdkFile>
vmkfstools -c 2048m testdisk1.vmdk
Initializing a Virtual Disk
vmkfstools -w <vmdkFile>
This option cleans the virtual disk by writing zeros over all its data. Depending on the size of your virtual disk and the I/O bandwidth to the device hosting the virtual disk, completing this command might take a long time.
Caution: When you use this command, you lose any existing data on the virtual disk.
Inflating a Thin Virtual Disk
vmkfstools -j <vmdkFile>
This option converts a thin virtual disk to eagerzeroedthick, preserving all existing data. The option allocates and zeroes out any blocks that are not already allocated.
Removing Zeroed Blocks (Converting a virtual disk to a thin disk)
vmkfstools -K <vmdkFile>
Use the vmkfstools command to convert any thin, zeroedthick, or eagerzeroedthick virtual disk to a thin disk with zeroed blocks removed.
This option deallocates all zeroed out blocks and leaves only those blocks that were allocated previously and contain valid data. The resulting virtual disk is in thin format.
Converting a Zeroedthick Virtual Disk to an Eagerzeroedthick Disk
vmkfstools -k <vmdkFile>
Use the vmkfstools command to convert any zeroedthick virtual disk to an eagerzeroedthick disk. While performing the conversion, this option preserves any data on the virtual disk.
Deleting a Virtual Disk
vmkfstools -U <vmdkFile>
This option deletes files associated with the virtual disk listed at the specified path on the VMFS volume.
Renaming a Virtual Disk
vmkfstools -E <oldName> <newName>
Cloning or Converting a Virtual Disk or Raw Disk
cloning: vmkfstools -i <sourceVmdkFile> <targetVmdkFile>
vmkfstools -i /vmfs/volumes/templates/gold-master.vmdk /vmfs/volumes/myVMFS/myOS.vmdk
converting: vmkfstools -i <sourceVmdkFile> -d <diskfomrat> <targetVmdkFile>
Extending a Virtual Disk
vmkfstools -X <newSize> [-d eagerzeroedthick] <vmdkFile>
use -d eagerzeroedthick to ensure the extended disk in eagerzeroedthick format.
Caution: do not extend the base disk of a virtual machine that has snapshots associated with it. If you do, you can no longer commit the snapshot or revert the base disk to its original size.
Displaying Virtual Disk Geometry
vmkfstools -g <vmdkFile>
The output is in the form: Geometry information C/H/S, where C represents the number of cylinders, H represents the number of heads, and S represents the number of sectors.
Checking and Repairing Virtual Disks
vmkfstools -x <vmdkFile>
Use this option to check or repair a virtual disk in case of an unclean shutdown

Storage Device Options

Managing SCSI Reservation of LUNs
Caution: Using the -L option can interrupt the operations of other servers on a SAN. Use the -L option only when troubleshooting clustering setups.
- vmkfstools -L reserve <deviceName>
  Reserves the specified LUN. After the reservation, only the server that reserved that LUN can access it. If other servers attempt to access that LUN, a reservation error results
- vmkfstools -L release <deviceName>
  Releases the reservation on the specified LUN. Other servers can access the LUN again
- vmkfstools -L lunreset <deviceName>
  Resets the specified LUN by clearing any reservation on the LUN and making the LUN available to all servers again. The reset does not affect any of the other LUNs on the device. If another LUN on the device is reserved, it remains reserved
- vmkfstools -L targetreset <deviceName>
  Resets the entire target. The reset clears any reservations on all the LUNs associated with that target and makes the LUNs available to all servers again.
- vmkfstools -L busrest <deviceName>
  Resets all accessible targets on the bus. The reset clears any reservation on all the LUNs accessible through the bus and makes them available to all servers agai
- When entering the device parameter, use the following format:
  /vmfs/devices/disks/vml.vml_ID:P

Hidden Options (reference: “Some useful vmkfstools ‘hidden’ options”)

VMDK Block Mappings
vmkfstools -t0 <vmdkFile>
Display the chuck file format in a VMDK file.

VMFS -- = eager zeroed thick
VMFS Z- = lazy zeroed thick
NOMP -- = thin

VSAN v6 Provision Thick Disk

I always think when creating or migrating VM on a VSAN datastore, its disk should be thin provisioned. However, I discovered some VM disks in our VSAN datastore are “thick” provisioned even all the VM storage policies are set to 0% object space reservation. How is it possible? After some digging, here is what I learn.

Thick Disk Format on VSAN

VSAN defines the disk type (thin or thick) via the Object Space Reservation setting in the VM Storage Policies. By default, this value is 0%, implying the disk is deployed as thin.

If the value is set to 100%, meaning the space for the disk is fully reserved, which can be thought of as full, thick provisioned. This behaves similarly to thick provision lazy zeroed. There is no eager-zeroed thick format on VSAN. (reference: Virtual SAN 6.2 Design and Sizing Guide, page 65)

Benefit to Provision Thick Disk on VSAN

Based on my understanding of VSAN disk IO operating (VSAN mirrors write IOs to all active mirrors, there are acknowledged when they hit the flash buffer!), typically there is no performance difference between thin and lazy zeroed thick provision on VSAN. Remember, there is no eager-zeroed thick format on VSAN (see above). Also see the Yellow-Bricks post. (PS: Duncan’s post may misspeak about VSAN eager zero thick provision.)

Provision Thick Disk on VSAN (Intentionally or By Accident)

There are several possible ways to provision a thick disk on VSAN.

Possibility #1

Define a thick VM Storage Policy
Set the Object Space Reservation to 100%
Use vSphere Web Client (cannot use vSphere C# Client)
Select the thick VM storage policy

Possibility #2

Use vSphere C# Client
Select “Thick Provision Lazy Zeroed” or “Thick Provision Eager Zeroed” on the disk type
I don’t know what the actual impact on VSAN when selecting eager zero. In my test, the VM disk is still created correctly. I will do more research and post an update.

Possibility #3

P2V a physical server to VM
By default, P2V uses thick provision on the disk
Change to Destination Disk to thin provision by select Advanced, Destination layout, Type, Thin

For VSAN 5.5, there is one more method, see here.

Change Thick Provisioned Disk to Thin on VSAN

Unfortunately, there is not a simple way to change a thick provisioned disk to thin on VSAN. Simply changing the VM storage policy on the disk has no impact.

In order to convert a thick disk to thin provisioned, do a storage migration of the disk to a SAN / NFS / local storage, then migrate back to the VSAN datastore. Make sure select the thin provision storage policy during the migration.

Brocade FC Switch FOS v7.2.0a WebTools Access in Windows Server 2012 R2 with IE 11

I got some errors (see at the end of the post) when setting up a brand new Brocade Fibre Channel switch running FOS v7.2.0a on a Windows Server 2012 R2 server with IE 11. The following instruction fixed the error.

Install Oracle JRE 1.7.0 update 25 Windows x86 version

According to its release note, FOS v7.2 is qualified and supported only with Oracle JRE 1.7.0 update 25.
Install JRE Windows x86 version (32-bit), instead of Windows x64 version (64-bit) even Windows Server 2012 R2 is a 64-bit OS

Launch “Java (32-bit)” in Control Panel

Security tab, lower Security Level to Medium
(optional) Advanced tab, set “Perform certificate revocation checks on” to “Do not check”. This will speed up the “Verifying application” process if the server does not have the Internet access.

Launch Internet Explorer

Click Tools, “Compatibility View settings” to add the Brocade switch IP address to the compatibility view list

Enter the IP address of the Brocade switch in Internet Explorer

The error messages I experienced and possible solutions

“The version of Java plugin needed to run the application is not installed. The page from where the plugin can be downloaded will be opened in a new window.”

Install the supported JRE version. see FOS release note for the supported JRE version
Install the 32-bit version of JRE, instead of 64-bit version
Verify Java Plug-In is enabled in IE
Add the FC switch IP to IE’s compatibility list

“Unable to launch the application” or “Unable to load resource: http://<switch-ip>/loc_res.jar

Install the supported JRE version. See FOS release note for the supported JRE version

“Application Blocked by Security Settings”

Lower the Java Security to Medium in Java 1.7 Update 25. For the newer version of Java, add the FC switch URL to the Java Security Exception Site List.

Create CD/DVD ISO File in Windows 10

Windows 8 or later has the built-in feature to open / mount ISO files. However, creating an ISO file requires other tools. There are many free utilities available.

Today, I tried ISO Recorder v.3.1.3 64-bit. The last update on its web support says it supports Windows Vista and 7. In my test, it works in Windows 10 as well.

Create an ISO file

There is no icon or shortcut to launch ISO Recorder after the installation. To create an ISO file, right-click on the CD / DVD drive, and select “Create image from CD/DVD”.

Mount an ISO file

Before installing ISO Recorder, Windows 10 will automatically mount the ISO to a virtual CD when double-clicking an ISO file. After installing ISO Recorder, double clicking an ISO file will launch ISO Recorder to write the image to a CD.

To mount the ISO file, right-click on the ISO file, and select “Open with” and “Windows Explorer”.

PS. I almost forget another free tool - ImgBurn. This was my go-to CD ISO creation and writing tool. The current version is v.2.5.8.0 released on June 16, 2013. This may still work in Windows 10, but I have not tried it yet.

Other free CD tools: MagicISO Virutal CD, WinISO 5.3, and CDBurnerXP.