Eddie's Blog

Extend Microsoft Cluster Shared Disk in VMware

A VM shared disk on Microsoft Cluster Service (MSCS) is running out of disk space. The VMs are on a single host (aka cluster in a box - CIB). I can think of two ways to expand the disk storage.

create a new big shared disk for the cluster, migrate the data, then change the new disk to the same drive letter as the original disk
extend the size of the existing shared disk

Obviously the latter seems simpler, but it requires special attention. The shared disk format in MSCS VMs must be in eager zeroed thick format. However, when extending an eagerzeroedthick VMDK, the extended chuck is in lazy zeroed thick format by default (reference “Extending an EagerZeroedThick Disk”. In my test, vSphere 6 has the same behavior)

Here is how I extend the MSCS shared disk

Power off both servers in the cluster
Increase the VMDK disk size. There are two ways:

GUI: edit the VM settings, increase the shared disk size
CLI: use vmkfstools -X <newsize> -d eagerzeroedthick <vmdkfile>

For more info about vmkfstools, see my vmkfstools Examples post

Using the GUI, the extended chuck will be in lazy zero thick format. The VM will fail to power on with the error “VMware ESX cannot open the virtual disk for clustering…”

There are two ways to convert the extended chuck to eagerzeroedthick format

Migrate the VM to another storage, and specify the eager zero thick format for the disk
Use vmkfstools -k <vmdkfile>

Once the entire shared disk is the eager zeroed thick format, the VM will be able to power on.
Extend the Windows partition as KB304736

vmkfstools Examples

When searching an issue on expanding a shared disk on Microsoft clustering VMs (CIB), I have learned more about the vmkfstools command.

The vmkfstools --help displays many options, but lack of explanation. So I document them here. (reference: vSphere Storage, Using vmkfstools)

# vmkfstools --help

OPTIONS FOR FILE SYSTEMS:

vmkfstools -C --createfs [vmfs3|vmfs5]
               -b --blocksize #[mMkK]
               -S --setfsname fsName
           -Z --spanfs span-partition
           -G --growfs grown-partition
   deviceName

           -P --queryfs -h --humanreadable
           -T --upgradevmfs
   vmfsPath
           -y --reclaimBlocks vmfsPath [--reclaimBlocksUnit #blocks]

OPTIONS FOR VIRTUAL DISKS:

vmkfstools -c --createvirtualdisk #[gGmMkK]
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               ]
               -a --adaptertype [buslogic|lsilogic|ide
                                |lsisas|pvscsi]
               -W --objecttype [file|vsan]
               --policyFile <fileName>
           -w --writezeros
           -j --inflatedisk
           -k --eagerzero
           -K --punchzero
           -U --deletevirtualdisk
           -E --renamevirtualdisk srcDisk
           -i --clonevirtualdisk srcDisk
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               |rdm:<device>|rdmp:<device>
                               |2gbsparse]
               -W --object [file|vsan]
               --policyFile <fileName>
               -N --avoidnativeclone
           -X --extendvirtualdisk #[gGmMkK]
               [-d --diskformat eagerzeroedthick]
           -M --migratevirtualdisk
           -r --createrdm /vmfs/devices/disks/...
           -q --queryrdm
           -z --createrdmpassthru /vmfs/devices/disks/...
           -v --verbose #
           -g --geometry
           -x --fix [check|repair]
           -e --chainConsistent
           -Q --objecttype name/value pair
           --uniqueblocks childDisk
   vmfsPath

OPTIONS FOR DEVICES:

vmkfstools -H --help

vmkfstools Command Syntax

vmkfstools options target

Options: separate into three types - File System Options, Virtual Disk Options, and Storage Device Options.
Target: partition, device, or path

File System Options

Listing Attributes of a VMFS Volume
The listed attributes include the file system label, if any, the number of extents comprising the specified VMFS volume, the UUID, and a listing of the device names where each extent resides.
vmkfstools -P -h <vmfsVolumePath>
vmkfstools -P -h /vmfs/volumes/netapp_sata_nfs1/
Creating a VMFS Datastore
vmkfstools -C vmfs5 -b <blocksize> -S <datastoreName> <partitionName>
vmkfstools -C vmfs5 -b 1m -S my_vmfs /vmfs/devices/disks/naa.ID:1
Extending an Existing VMFS Volume
vmkfstools -Z <span_partition> <head_partition>
vmkfstools -Z /vmfs/devices/disks/naa.disk_ID_2:1 /vmfs/devices/disks/naa.disk_ID_1:1
Caution: When you run this option, you lose all data that previously existed on the SCSI device you specified in span_partition.
Growing an Existing Extent
vmkfstools –G device device
vmkfstools --growfs /vmfs/devices/disks/disk_ID:1 /vmfs/devices/disks/disk_ID:1

Virtual Disk Options

Creating a Virtual Disk
vmkfstools -c <size> -d <diskformat> <vmdkFile>
vmkfstools -c 2048m testdisk1.vmdk
Initializing a Virtual Disk
vmkfstools -w <vmdkFile>
This option cleans the virtual disk by writing zeros over all its data. Depending on the size of your virtual disk and the I/O bandwidth to the device hosting the virtual disk, completing this command might take a long time.
Caution: When you use this command, you lose any existing data on the virtual disk.
Inflating a Thin Virtual Disk
vmkfstools -j <vmdkFile>
This option converts a thin virtual disk to eagerzeroedthick, preserving all existing data. The option allocates and zeroes out any blocks that are not already allocated.
Removing Zeroed Blocks (Converting a virtual disk to a thin disk)
vmkfstools -K <vmdkFile>
Use the vmkfstools command to convert any thin, zeroedthick, or eagerzeroedthick virtual disk to a thin disk with zeroed blocks removed.
This option deallocates all zeroed out blocks and leaves only those blocks that were allocated previously and contain valid data. The resulting virtual disk is in thin format.
Converting a Zeroedthick Virtual Disk to an Eagerzeroedthick Disk
vmkfstools -k <vmdkFile>
Use the vmkfstools command to convert any zeroedthick virtual disk to an eagerzeroedthick disk. While performing the conversion, this option preserves any data on the virtual disk.
Deleting a Virtual Disk
vmkfstools -U <vmdkFile>
This option deletes files associated with the virtual disk listed at the specified path on the VMFS volume.
Renaming a Virtual Disk
vmkfstools -E <oldName> <newName>
Cloning or Converting a Virtual Disk or Raw Disk
cloning: vmkfstools -i <sourceVmdkFile> <targetVmdkFile>
vmkfstools -i /vmfs/volumes/templates/gold-master.vmdk /vmfs/volumes/myVMFS/myOS.vmdk
converting: vmkfstools -i <sourceVmdkFile> -d <diskfomrat> <targetVmdkFile>
Extending a Virtual Disk
vmkfstools -X <newSize> [-d eagerzeroedthick] <vmdkFile>
use -d eagerzeroedthick to ensure the extended disk in eagerzeroedthick format.
Caution: do not extend the base disk of a virtual machine that has snapshots associated with it. If you do, you can no longer commit the snapshot or revert the base disk to its original size.
Displaying Virtual Disk Geometry
vmkfstools -g <vmdkFile>
The output is in the form: Geometry information C/H/S, where C represents the number of cylinders, H represents the number of heads, and S represents the number of sectors.
Checking and Repairing Virtual Disks
vmkfstools -x <vmdkFile>
Use this option to check or repair a virtual disk in case of an unclean shutdown

Storage Device Options

Managing SCSI Reservation of LUNs
Caution: Using the -L option can interrupt the operations of other servers on a SAN. Use the -L option only when troubleshooting clustering setups.
- vmkfstools -L reserve <deviceName>
  Reserves the specified LUN. After the reservation, only the server that reserved that LUN can access it. If other servers attempt to access that LUN, a reservation error results
- vmkfstools -L release <deviceName>
  Releases the reservation on the specified LUN. Other servers can access the LUN again
- vmkfstools -L lunreset <deviceName>
  Resets the specified LUN by clearing any reservation on the LUN and making the LUN available to all servers again. The reset does not affect any of the other LUNs on the device. If another LUN on the device is reserved, it remains reserved
- vmkfstools -L targetreset <deviceName>
  Resets the entire target. The reset clears any reservations on all the LUNs associated with that target and makes the LUNs available to all servers again.
- vmkfstools -L busrest <deviceName>
  Resets all accessible targets on the bus. The reset clears any reservation on all the LUNs accessible through the bus and makes them available to all servers agai
- When entering the device parameter, use the following format:
  /vmfs/devices/disks/vml.vml_ID:P

Hidden Options (reference: “Some useful vmkfstools ‘hidden’ options”)

VMDK Block Mappings
vmkfstools -t0 <vmdkFile>
Display the chuck file format in a VMDK file.

VMFS -- = eager zeroed thick
VMFS Z- = lazy zeroed thick
NOMP -- = thin

VSAN v6 Provision Thick Disk

I always think when creating or migrating VM on a VSAN datastore, its disk should be thin provisioned. However, I discovered some VM disks in our VSAN datastore are “thick” provisioned even all the VM storage policies are set to 0% object space reservation. How is it possible? After some digging, here is what I learn.

Thick Disk Format on VSAN

VSAN defines the disk type (thin or thick) via the Object Space Reservation setting in the VM Storage Policies. By default, this value is 0%, implying the disk is deployed as thin.

If the value is set to 100%, meaning the space for the disk is fully reserved, which can be thought of as full, thick provisioned. This behaves similarly to thick provision lazy zeroed. There is no eager-zeroed thick format on VSAN. (reference: Virtual SAN 6.2 Design and Sizing Guide, page 65)

Benefit to Provision Thick Disk on VSAN

Based on my understanding of VSAN disk IO operating (VSAN mirrors write IOs to all active mirrors, there are acknowledged when they hit the flash buffer!), typically there is no performance difference between thin and lazy zeroed thick provision on VSAN. Remember, there is no eager-zeroed thick format on VSAN (see above). Also see the Yellow-Bricks post. (PS: Duncan’s post may misspeak about VSAN eager zero thick provision.)

Provision Thick Disk on VSAN (Intentionally or By Accident)

There are several possible ways to provision a thick disk on VSAN.

Possibility #1

Define a thick VM Storage Policy
Set the Object Space Reservation to 100%
Use vSphere Web Client (cannot use vSphere C# Client)
Select the thick VM storage policy

Possibility #2

Use vSphere C# Client
Select “Thick Provision Lazy Zeroed” or “Thick Provision Eager Zeroed” on the disk type
I don’t know what the actual impact on VSAN when selecting eager zero. In my test, the VM disk is still created correctly. I will do more research and post an update.

Possibility #3

P2V a physical server to VM
By default, P2V uses thick provision on the disk
Change to Destination Disk to thin provision by select Advanced, Destination layout, Type, Thin

For VSAN 5.5, there is one more method, see here.

Change Thick Provisioned Disk to Thin on VSAN

Unfortunately, there is not a simple way to change a thick provisioned disk to thin on VSAN. Simply changing the VM storage policy on the disk has no impact.

In order to convert a thick disk to thin provisioned, do a storage migration of the disk to a SAN / NFS / local storage, then migrate back to the VSAN datastore. Make sure select the thin provision storage policy during the migration.

Brocade FC Switch FOS v7.2.0a WebTools Access in Windows Server 2012 R2 with IE 11

I got some errors (see at the end of the post) when setting up a brand new Brocade Fibre Channel switch running FOS v7.2.0a on a Windows Server 2012 R2 server with IE 11. The following instruction fixed the error.

Install Oracle JRE 1.7.0 update 25 Windows x86 version

According to its release note, FOS v7.2 is qualified and supported only with Oracle JRE 1.7.0 update 25.
Install JRE Windows x86 version (32-bit), instead of Windows x64 version (64-bit) even Windows Server 2012 R2 is a 64-bit OS

Launch “Java (32-bit)” in Control Panel

Security tab, lower Security Level to Medium
(optional) Advanced tab, set “Perform certificate revocation checks on” to “Do not check”. This will speed up the “Verifying application” process if the server does not have the Internet access.

Launch Internet Explorer

Click Tools, “Compatibility View settings” to add the Brocade switch IP address to the compatibility view list

Enter the IP address of the Brocade switch in Internet Explorer

The error messages I experienced and possible solutions

“The version of Java plugin needed to run the application is not installed. The page from where the plugin can be downloaded will be opened in a new window.”

Install the supported JRE version. see FOS release note for the supported JRE version
Install the 32-bit version of JRE, instead of 64-bit version
Verify Java Plug-In is enabled in IE
Add the FC switch IP to IE’s compatibility list

“Unable to launch the application” or “Unable to load resource: http://<switch-ip>/loc_res.jar

Install the supported JRE version. See FOS release note for the supported JRE version

“Application Blocked by Security Settings”

Lower the Java Security to Medium in Java 1.7 Update 25. For the newer version of Java, add the FC switch URL to the Java Security Exception Site List.

Create CD/DVD ISO File in Windows 10

Windows 8 or later has the built-in feature to open / mount ISO files. However, creating an ISO file requires other tools. There are many free utilities available.

Today, I tried ISO Recorder v.3.1.3 64-bit. The last update on its web support says it supports Windows Vista and 7. In my test, it works in Windows 10 as well.

Create an ISO file

There is no icon or shortcut to launch ISO Recorder after the installation. To create an ISO file, right-click on the CD / DVD drive, and select “Create image from CD/DVD”.

Mount an ISO file

Before installing ISO Recorder, Windows 10 will automatically mount the ISO to a virtual CD when double-clicking an ISO file. After installing ISO Recorder, double clicking an ISO file will launch ISO Recorder to write the image to a CD.

To mount the ISO file, right-click on the ISO file, and select “Open with” and “Windows Explorer”.

PS. I almost forget another free tool - ImgBurn. This was my go-to CD ISO creation and writing tool. The current version is v.2.5.8.0 released on June 16, 2013. This may still work in Windows 10, but I have not tried it yet.

Other free CD tools: MagicISO Virutal CD, WinISO 5.3, and CDBurnerXP.

Do Not Upgrade Dell Server with H730 and FD332-PERC Controller to VSAN 6.2

VMware released VSAN 6.2 on March 15, 2016. However, if your VSAN is running on a Dell server with H730 or FD332-PERC controller, do not upgrade to VSAN 6.2.

See KB2144614 for more information.

My IOPS Calculator

I look for an IOPS calculator for the EMC VNX2 5400 storage that I am working on. There are many available on the Internet. But none of them gives me exactly what I want. More frustrated, different calculators produce different results. So I decide to build one myself. Here is what I get.

My IOPS calculator concept:

Based on EMC white paper “EMC VNX2 Unified Best Practices for Performance, October 2015”.
Use EMC’s recommended RAID type configuration, IOPS, and formula.
Use formula: Raw IOPS = Actual IOPS * Read % + Actual IOPS * Write % * RAID Penalty
- Raw IOPS = Disk IOPS, Back-end IOPS
- Actual IOPS = Host IOPS, Front-end IOPS
Some calculators use different formula (see discussion). I can’t say they are wrong, but EMC’s formula makes more sense to me. I list all the ones I tested here
- Result matches my calculator
- Result not match my calculator
There are two calculators in my spreadsheet
- From disk qty, size, type, RAID type, and read/write percentage to get the actual usable storage and actual IOPS
- From actual IOPS, disk size, type, RAID type, and read/write percentage to get the required disk qty

My IOPS Calculator Download (save the spreadsheet and open in Excel)