Extend Microsoft Cluster Shared Disk in VMware

A VM shared disk on Microsoft Cluster Service (MSCS) is running out of disk space. The VMs are on a single host (aka cluster in a box - CIB). I can think of two ways to expand the disk storage.

• create a new big shared disk for the cluster, migrate the data, then change the new disk to the same drive letter as the original disk
• extend the size of the existing shared disk

Obviously the latter seems simpler, but it requires special attention. The shared disk format in MSCS VMs must be in eager zeroed thick format. However, when extending an eagerzeroedthick VMDK, the extended chuck is in lazy zeroed thick format by default (reference “Extending an EagerZeroedThick Disk”. In my test, vSphere 6 has the same behavior)

Here is how I extend the MSCS shared disk

• Power off both servers in the cluster
• Increase the VMDK disk size. There are two ways:
• GUI: edit the VM settings, increase the shared disk size
• CLI: use vmkfstools -X <newsize> -d eagerzeroedthick <vmdkfile>
• For more info about vmkfstools, see my vmkfstools Examples post
• Using the GUI, the extended chuck will be in lazy zero thick format. The VM will fail to power on with the error “VMware ESX cannot open the virtual disk for clustering…”

• There are two ways to convert the extended chuck to eagerzeroedthick format
• Migrate the VM to another storage, and specify the eager zero thick format for the disk
• Use vmkfstools -k <vmdkfile>
  
• Once the entire shared disk is the eager zeroed thick format, the VM will be able to power on.
• Extend the Windows partition as KB304736

vmkfstools Examples

When searching an issue on expanding a shared disk on Microsoft clustering VMs (CIB), I have learned more about the vmkfstools command.

The vmkfstools --help displays many options, but lack of explanation. So I document them here. (reference: vSphere Storage, Using vmkfstools)

# vmkfstools --help

OPTIONS FOR FILE SYSTEMS:

vmkfstools -C --createfs [vmfs3|vmfs5]
               -b --blocksize #[mMkK]
               -S --setfsname fsName
           -Z --spanfs span-partition
           -G --growfs grown-partition
   deviceName

           -P --queryfs -h --humanreadable
           -T --upgradevmfs
   vmfsPath
           -y --reclaimBlocks vmfsPath [--reclaimBlocksUnit #blocks]

OPTIONS FOR VIRTUAL DISKS:

vmkfstools -c --createvirtualdisk #[gGmMkK]
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               ]
               -a --adaptertype [buslogic|lsilogic|ide
                                |lsisas|pvscsi]
               -W --objecttype [file|vsan]
               --policyFile <fileName>
           -w --writezeros
           -j --inflatedisk
           -k --eagerzero
           -K --punchzero
           -U --deletevirtualdisk
           -E --renamevirtualdisk srcDisk
           -i --clonevirtualdisk srcDisk
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               |rdm:<device>|rdmp:<device>
                               |2gbsparse]
               -W --object [file|vsan]
               --policyFile <fileName>
               -N --avoidnativeclone
           -X --extendvirtualdisk #[gGmMkK]
               [-d --diskformat eagerzeroedthick]
           -M --migratevirtualdisk
           -r --createrdm /vmfs/devices/disks/...
           -q --queryrdm
           -z --createrdmpassthru /vmfs/devices/disks/...
           -v --verbose #
           -g --geometry
           -x --fix [check|repair]
           -e --chainConsistent
           -Q --objecttype name/value pair
           --uniqueblocks childDisk
   vmfsPath

OPTIONS FOR DEVICES:

           -L --lock [reserve|release|lunreset|targetreset|busreset|readkeys|readresv
                     ] /vmfs/devices/disks/...
           -B --breaklock /vmfs/devices/disks/...

vmkfstools -H --help

vmkfstools Command Syntax

vmkfstools options target

Options: separate into three types - File System Options, Virtual Disk Options, and Storage Device Options.
Target: partition, device, or path

File System Options

• Listing Attributes of a VMFS Volume
  The listed attributes include the file system label, if any, the number of extents comprising the specified VMFS volume, the UUID, and a listing of the device names where each extent resides.
  vmkfstools -P -h <vmfsVolumePath>
  vmkfstools -P -h /vmfs/volumes/netapp_sata_nfs1/
• Creating a VMFS Datastore
  vmkfstools -C vmfs5 -b <blocksize> -S <datastoreName> <partitionName>
  vmkfstools -C vmfs5 -b 1m -S my_vmfs /vmfs/devices/disks/naa.ID:1
• Extending an Existing VMFS Volume
  vmkfstools -Z <span_partition> <head_partition>
  vmkfstools -Z /vmfs/devices/disks/naa.disk_ID_2:1 /vmfs/devices/disks/naa.disk_ID_1:1
  Caution: When you run this option, you lose all data that previously existed on the SCSI device you specified in span_partition.
• Growing an Existing Extent
  vmkfstools –G device device
  vmkfstools --growfs /vmfs/devices/disks/disk_ID:1 /vmfs/devices/disks/disk_ID:1

Virtual Disk Options

• Creating a Virtual Disk
  vmkfstools -c <size> -d <diskformat> <vmdkFile>
  vmkfstools -c 2048m testdisk1.vmdk
• Initializing a Virtual Disk
  vmkfstools -w <vmdkFile>
  This option cleans the virtual disk by writing zeros over all its data. Depending on the size of your virtual disk and the I/O bandwidth to the device hosting the virtual disk, completing this command might take a long time.
  Caution: When you use this command, you lose any existing data on the virtual disk.
• Inflating a Thin Virtual Disk
  vmkfstools -j <vmdkFile>
  This option converts a thin virtual disk to eagerzeroedthick, preserving all existing data. The option allocates and zeroes out any blocks that are not already allocated.
• Removing Zeroed Blocks (Converting a virtual disk to a thin disk)
  vmkfstools -K <vmdkFile>
  Use the vmkfstools command to convert any thin, zeroedthick, or eagerzeroedthick virtual disk to a thin disk with zeroed blocks removed.
  This option deallocates all zeroed out blocks and leaves only those blocks that were allocated previously and contain valid data. The resulting virtual disk is in thin format.
• Converting a Zeroedthick Virtual Disk to an Eagerzeroedthick Disk
  vmkfstools -k <vmdkFile>
  Use the vmkfstools command to convert any zeroedthick virtual disk to an eagerzeroedthick disk. While performing the conversion, this option preserves any data on the virtual disk.
• Deleting a Virtual Disk
  vmkfstools -U <vmdkFile>
  This option deletes files associated with the virtual disk listed at the specified path on the VMFS volume.
• Renaming a Virtual Disk
  vmkfstools -E <oldName> <newName>
• Cloning or Converting a Virtual Disk or Raw Disk
  cloning: vmkfstools -i <sourceVmdkFile> <targetVmdkFile>
  vmkfstools -i /vmfs/volumes/templates/gold-master.vmdk /vmfs/volumes/myVMFS/myOS.vmdk
  converting: vmkfstools -i <sourceVmdkFile> -d <diskfomrat> <targetVmdkFile>
• Extending a Virtual Disk
  vmkfstools -X <newSize> [-d eagerzeroedthick] <vmdkFile>
  use -d eagerzeroedthick to ensure the extended disk in eagerzeroedthick format.
  Caution: do not extend the base disk of a virtual machine that has snapshots associated with it. If you do, you can no longer commit the snapshot or revert the base disk to its original size.
• Displaying Virtual Disk Geometry
  vmkfstools -g <vmdkFile>
  The output is in the form: Geometry information C/H/S, where C represents the number of cylinders, H represents the number of heads, and S represents the number of sectors.

• Checking and Repairing Virtual Disks
  vmkfstools -x <vmdkFile>
  Use this option to check or repair a virtual disk in case of an unclean shutdown

Storage Device Options

• Managing SCSI Reservation of LUNs
  Caution: Using the -L option can interrupt the operations of other servers on a SAN. Use the -L option only when troubleshooting clustering setups.
  • vmkfstools -L reserve <deviceName>
    Reserves the specified LUN. After the reservation, only the server that reserved that LUN can access it. If other servers attempt to access that LUN, a reservation error results
  • vmkfstools -L release <deviceName>
    Releases the reservation on the specified LUN. Other servers can access the LUN again
  • vmkfstools -L lunreset <deviceName>
    Resets the specified LUN by clearing any reservation on the LUN and making the LUN available to all servers again. The reset does not affect any of the other LUNs on the device. If another LUN on the device is reserved, it remains reserved
  • vmkfstools -L targetreset <deviceName>
    Resets the entire target. The reset clears any reservations on all the LUNs associated with that target and makes the LUNs available to all servers again.
  • vmkfstools -L busrest <deviceName>
    Resets all accessible targets on the bus. The reset clears any reservation on all the LUNs accessible through the bus and makes them available to all servers agai
  • When entering the device parameter, use the following format:
    /vmfs/devices/disks/vml.vml_ID:P

Hidden Options (reference: “Some useful vmkfstools ‘hidden’ options”)

• VMDK Block Mappings
  vmkfstools -t0 <vmdkFile>
  Display the chuck file format in a VMDK file.
• VMFS -- = eager zeroed thick
• VMFS Z- = lazy zeroed thick
• NOMP -- = thin

Brocade FC Switch FOS v7.2.0a WebTools Access in Windows Server 2012 R2 with IE 11

I got some errors (see at the end of the post) when setting up a brand new Brocade Fibre Channel switch running FOS v7.2.0a on a Windows Server 2012 R2 server with IE 11. The following instruction fixed the error.

• Install Oracle JRE 1.7.0 update 25 Windows x86 version
• According to its release note, FOS v7.2 is qualified and supported only with Oracle JRE 1.7.0 update 25.
• Install JRE Windows x86 version (32-bit), instead of Windows x64 version (64-bit) even Windows Server 2012 R2 is a 64-bit OS
• Launch “Java (32-bit)” in Control Panel
• Security tab, lower Security Level to Medium
• 
• (optional) Advanced tab, set “Perform certificate revocation checks on” to “Do not check”. This will speed up the “Verifying application” process if the server does not have the Internet access.
• 
• Launch Internet Explorer
• Click Tools, “Compatibility View settings” to add the Brocade switch IP address to the compatibility view list
• 
• 
• Enter the IP address of the Brocade switch in Internet Explorer
• 
• 
• 

The error messages I experienced and possible solutions

• “The version of Java plugin needed to run the application is not installed. The page from where the plugin can be downloaded will be opened in a new window.”
• Install the supported JRE version. see FOS release note for the supported JRE version
• Install the 32-bit version of JRE, instead of 64-bit version
• Verify Java Plug-In is enabled in IE
• Add the FC switch IP to IE’s compatibility list
• “Unable to launch the application” or “Unable to load resource: http://<switch-ip>/loc_res.jar
• Install the supported JRE version. See FOS release note for the supported JRE version
• “Application Blocked by Security Settings”
• Lower the Java Security to Medium in Java 1.7 Update 25. For the newer version of Java, add the FC switch URL to the Java Security Exception Site List.

My IOPS Calculator

I look for an IOPS calculator for the EMC VNX2 5400 storage that I am working on. There are many available on the Internet. But none of them gives me exactly what I want. More frustrated, different calculators produce different results. So I decide to build one myself. Here is what I get.

My IOPS calculator concept:

• Based on EMC white paper “EMC VNX2 Unified Best Practices for Performance, October 2015”.
• Use EMC’s recommended RAID type configuration, IOPS, and formula.
• Use formula: Raw IOPS = Actual IOPS * Read % + Actual IOPS * Write % * RAID Penalty
  • Raw IOPS = Disk IOPS, Back-end IOPS
  • Actual IOPS = Host IOPS, Front-end IOPS
• Some calculators use different formula (see discussion). I can’t say they are wrong, but EMC’s formula makes more sense to me. I list all the ones I tested here
  • Result matches my calculator
    • http://www.yellow-bricks.com/2009/12/23/iops/
    • http://wintelguy.com/raidperf.pl
    • http://www.vclouds.nl/calculate-iops/
    • http://www.ryanfrantz.com/posts/calculating-disk-iops/
  • Result not match my calculator
    • http://thecloudcalculator.com/calculators/disk-raid-and-iops.html
    • http://www.myvmwareblog.com/2012/12/29/iops-calculator-2/
    • http://www.eksendata.com/araclar/iops/iopse.php
• There are two calculators in my spreadsheet
  • From disk qty, size, type, RAID type, and read/write percentage to get the actual usable storage and actual IOPS
  • From actual IOPS, disk size, type, RAID type, and read/write percentage to get the required disk qty

 

My IOPS Calculator Download (save the spreadsheet and open in Excel)

Fix A SAN Datastore Inaccessible On A ESXi Host

A SAN datastore is shown inaccessible on one of the ESXi hosts in the cluster. Other ESXi hosts can access that datastore without problem.

Solution: restart the ESXi management agents on the ESXi host

There are a few ways to restart the management agents (KB1003490).

• From the Direct Console User Interface (DCUI)
• Press F2 to customize the system
• Log in as root
• Under Troubleshooting Options, select Restart Management Agents
• From the Local Console (Alt + F2) or SSH
• Log in as root
• run these commands
• /etc/init.d/hostd restart
• /etc/init.d/vpxa restart
• if the hostd is not restart, use KB1005566 to find and kill hostd Process ID (PID), then start it again (/etc/init.d/hostd start)
• alternatively
• To reset the management network on a specific VMkernel interface, by default vmk0
• esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0
• Note: run the above commands together, using a semicolon (;) between the two commands
• To restart all management agents on the host
• services.sh restart
• Caution:
• check if LACP is enabled on the VDS’s Uplink Port Group
• If LACP is not configured, the services.sh script can be safely executed
• If LACP is enabled and configured, do not restart management services using services.sh. Instead restart independent services using /etc/init.d/hostd restart and /etc/init.d/vpxa restart.
• If the issue is not resolved, take a downtime before restarting all services with services.sh