Fix A SAN Datastore Inaccessible On A ESXi Host

A SAN datastore is shown inaccessible on one of the ESXi hosts in the cluster. Other ESXi hosts can access that datastore without problem.

Solution: restart the ESXi management agents on the ESXi host

There are a few ways to restart the management agents (KB1003490).

• From the Direct Console User Interface (DCUI)
• Press F2 to customize the system
• Log in as root
• Under Troubleshooting Options, select Restart Management Agents
• From the Local Console (Alt + F2) or SSH
• Log in as root
• run these commands
• /etc/init.d/hostd restart
• /etc/init.d/vpxa restart
• if the hostd is not restart, use KB1005566 to find and kill hostd Process ID (PID), then start it again (/etc/init.d/hostd start)
• alternatively
• To reset the management network on a specific VMkernel interface, by default vmk0
• esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0
• Note: run the above commands together, using a semicolon (;) between the two commands
• To restart all management agents on the host
• services.sh restart
• Caution:
• check if LACP is enabled on the VDS’s Uplink Port Group
• If LACP is not configured, the services.sh script can be safely executed
• If LACP is enabled and configured, do not restart management services using services.sh. Instead restart independent services using /etc/init.d/hostd restart and /etc/init.d/vpxa restart.
• If the issue is not resolved, take a downtime before restarting all services with services.sh

Backup Consistency Types

This post is to summarize the various backup consistency types:

• Inconsistent Backup
  • Any file changed after it was backed up but before the job completed, the result is an inconsistent backup
  • e.g. File A and B, File A is backed up, then File A and B are changed, then File B is backed up. Now the backup of File A and B is inconsistent
  • The content in memory or pending I/O is not backed up
• Crash-Consistent Backup
  • All data is backed up at exactly the same time via techniques like Volume Shadow Copy Service (VSS) to take a block-level snapshot, and then the backup software pulls its backup from that snapshot
  • This backed up data is in the same state it would have been if the system had crashed
  • The content in memory or pending I/O is not backed up
  • Many applications, like Active Directory, have an automated recovery mechanism and will attempt to handle the inconsistent problem without administrator intervention. If these automated systems aren’t successful, a manual process is needed. For Microsoft SQL, you may need to know how to replay logs into a database file.
• Application-Consistent Backup
  • For Windows applications, the application manufacturer provide a VSS writer. When the VSS service is triggered, it will notify these writers that a backup is occuring. Then it’s up to the VSS writer how to handle it.
  • A proper VSS writer will make the application flushing all of its memory and I/O operations to the disk, as it would be if the application were properly closed
  • When the VSS snapshot is complete, it signals the VSS writers, then application resume normal operation and the backup software pulls its backup from that snapshot
  • If an application does not provide or properly register a VSS provider but its data resides on a volume with VSS enabled, the data is backed up in a crash-consistent state
• Image-Level Backup
  • The other backups back up when a machine is actively running
  • An image-level backup backs up when the machine is shut down

Source: VSS Crash-Consistent vs Application-Consistent VSS Backups

vMotion Microsoft Cluster VMs

vSphere supports three different configurations of Microsoft Cluster Service (MSCS):

• Clustering MSCS VMs on a single host (aka a cluster in a box - CIB)
• Clustering MSCS VMs across physical hosts (aka a cluster across boxes - CAB)
• Clustering physical machines with VM

see Setup for Failover Clustering and Microsoft Cluster Service, ESXi 6.0 for more information.

However, vMotion is supported only for CAB with pass-through RDMs. Do not vMotion MSCS VMs in the other two configuration.

In addition, do not vMotion MSCS VMs to a VSAN storage, because VSAN does not support thick provision and SCSI bus sharing on the VM SCSI adapter. The VM will not be able to power on with the error message “Cannot use non-thick disks with clustering enabled (sharedBus='physical'). The disk for scsi1:0 is of the type thin.”

Add Fonts to Windows Command Prompt

By default, Windows 10 Command Prompt only offers two font choices - Consolas and Lucida Console, even other fonts are installed. To add more a font to selection, add a String Value registry key in

HKLM\Software\Microsoft\WindowsNT\CurrentVersion\Console\TrueTypeFont

In the following screenshot, I added Bitstream Vera Sans Mono to my command prompt.

Source: http://www.watchingthenet.com/how-to-add-and-change-fonts-in-windows-command-prompt.html

Multiple vCPUs Can Cause Performance Issues

Assigning more hardware resource (aka overprovisioning) on a VM can cause performance issues.

According to KB1005362, we can use esxtop to check the %CSTP value to determine vCPUs overprovisioning. If the value of a VM is higher than 3.00, the performance issue may be caused by the vCPU count. Try lowering the vCPU count of the VM by 1.

Understand Three Important Files in vSphere HA

This post is the note I take as I am reading the vSphere 6.x HA Deepdive book, plus my understanding of the materials.

Here we focuse on the files stored on the shared datastore, aka. remote files. Each host also restores some configuration files in a directly accessible datastore, aka. local files)

• Protectedlist file
  • Naming: protectedlist
  • Owner: The master locks this protectedlist file
  • The master uses this file to claim the “ownership” of the datastores stored the VM configuration file. When a host is isolated, if the host can access the datastore, then it will validate whether a master owns the datastores. If no master owns the datastores, the isolation response will not be triggered and restarts will not be initiated. (see page 29 & 30 of the book about isolation response).
  • The master uses this protectedlist file to track the VMs protected by HA and the states of the VMs (powered on / off)
  • The master distributes this protectedlist file to all datastores in use by the VMs in the cluster
• "poweron" file
  • Naming: host-<number>-poweron
  • Owner: per-host (master & slaves)
  • The host uses this "poweron" file to track the powered on virtual machines on a host
  • The slaves use this "poweron" file to inform the master that it is isolated from the management network
    • No datastore heartbeat: the master determines a host has failed
    • The top line of "poweron" file is 1 (means isolated); if 0 means not-isolated
• Heartbeat file
  • Naming: host-<number>-hb
  • Owner: per-host
  • Each host creates a heartbeat file on the designed heartbeating datastores
    • On VMFS datastore, "heartbeat region" is used to check the heartbeat update
    • On NFS database, the time-stamp of the file is check (each host writes to its heartbeat file once every 5 seconds)

Disable Mac Startup Sound

A way to disable the Mac startup sound is to mute the speaker before shutting down the computer. But you have to remember to do that every time you unmute the speaker.

When I search on the web, some third party utilities are mentioned, e.g. StartupSound.prefPane or Psst or StartNinja or use sudo nvram SystemAudioVolume=%80. All these utilities are for the older version of Mac; and the nvram command doesn’t work my OS X El Capitan (version 10.11.3).

Finally, I find a way to mute the speaker automatically when log off. See this post for the full detail. I only set up the mute.sh, and don’t even bother the unmute.sh.

1. create a mute.sh script with the following lines
• #!/bin/bash
• osascript -e ‘set volume with output muted’
2. sudo chmod u+x ~/Documents/mute.sh
3. sudo mv ~/Document/mute.sh /Library/Scripts/
4. sudo defaults write com.apple.loginwindow LogoutHook /Library/Scripts/mute.sh

To restore the startup sound

• sudo defaults delete com.apple.loginwindow LogoutHook