Search This Blog

VUM 6.5 “Cannot download patch definitions” via UMDS 6.5 Work Around

My nested vSphere lab environment does not have the access to the Internet (there is no physical network adapter as the uplink on the lab port group). To update and patch the ESXi host using vSphere Update Manager (VUM), I installed Update Manager Download Service (UMDS) on a Windows Server VM with dual NICs - one for Internet, another for the lab port group. Use the UMDS to download the update, configure IIS as the web server for the update repository, and configure VUM to use the http share repository. It worked fine in vSphere 6.0 and 6.2.

Recently I upgraded the lab environment to vSphere 6.5. The vCenter Server Appliance and ESXi hosts are upgraded to 6.5 successfully.

vCenter Server Appliance 6.5 bundles with VUM. VUM no longer requires a Windows Server. For UMDS, it can be installed on a Windows or Linux server. Since I already have a Windows server for UMDS. I continue using it instead of setting up a Linux server. But UMDS can’t be upgraded from the previous version to 6.5.

I uninstalled UMDS 6.0 and SQL Server 2012 Express on the server, and installed UMDS 6.5 with SQL Server 2012 Express from the ISO. I used the UMDS 6.0 repository folder for UMDS 6.5 and configured UMDS to download the host update only. Since IIS is already set up and I used the same repository folder, no change is needed in IIS. UMDS 6.5 successfully downloaded the update files from VMware.

I configured the VUM 6.5 to use the IIS shared repository. VUM validated the URL successfully. However, when I clicked “Download Now” button in VUM, I got the “Cannot download patch definitions” error.

Troubleshooting

umds.missing.6.5.metadata

Solution

  • A similar issue happened in vSphere 5.5 (KB2061622)
  • I can’t find a patch available for ESXi 6.5 yet, but I can apply the similar work around by removing <metadata> reference for vmw-ESXi-6.5.0-metadata.zip in
    hostupdate\vmw\__hostupdate20-consolidated-metadata-index__.xml
  • Now the download patch definition is successful when I click “Download Now” button in VUM
  • However, running UMDS again will re-add the vmw-ESXi-6.5.0-metadata.zip reference, and VUM will fail to download the patch definition
  • This may be a bug in vSphere 6.5 (like vSphere 5.5). Hope VMware can fix in the next update

Stop A Task Stuck in vCenter Server Appliance

  1. find out the name of vSphere host running the stuck task if possible
  2. SSH to the vCenter Server server appliance
  3. service vmware-vpxd restart
    • After restart the vmware-vpxd service on the vCSA, the stuck task should disappear from the vSphere web client
    • However, the task may be still running on the vSphere host
    • Use the following steps to stop the stuck task on the vSphere host
  4. SSH to the vSphere host running the task
  5. /etc/init.d/hostd restart
  6. /etc/init.d/vpxa restart

VSAN 6.2 On-disk Format Upgrade Fails at 5%

I am working on upgrading our VSAN from 6.1 to 6.2. See this from the upgrade step overview.

After upgrading each VSAN host to ESXi 6.0U2 (the latest build 4510822 as of 11/01/2016), the last step is to upgrade the on-disk format from v2 to v3.

In our case, the on-disk format upgrade fails at 5% with the error message “General Virtual SAN error. Disk Format conversion failed due to unexpected error”.

vsan.6.2.on-disk.format.upgrade.fail.at.5%.01

However, check the disk format in VSAN cluster, Manage, Settings, Virtual SAN / Disk Management. A disk group is upgraded to the interim version 2.5 each time I run the on-disk format upgrade. In the screen shots below, I ran the on-disk format upgrade twice. Two of the disk groups are upgraded to v2.5.

vsan.6.2.on-disk.format.upgrade.fail.at.5%.02

I keep running the on-disk format upgrade. In our VSAN, we have 4 hosts with 2 disk groups on each node. The on-disk format failed six times. On the seventh time, all disk groups are upgraded to v2.5.

vsan.6.2.on-disk.format.upgrade.fail.at.5%.03

Then the upgrade moves forward to the next process - starting remove disks from one of the VSAN host.

vsan.6.2.on-disk.format.upgrade.fail.at.5%.04

I have not figured out the cause of the failure. Re-running the upgrade process until all the disk groups are upgraded to the format v2.5 is able to keep the process moving forward.

VMware Tools Stuck in “Upgrade in progress” Fix

I notice the VMware Tools status on some VMs (mostly Linux) is “Upgrade in progress”.

vmware.tools.upgrade.in.progress.01

For these VMs, I cannot vMotion them to another host; and some of these VMs, the “Edit Settings” and “Edit Resource Settings” are grayed out.

Solution:

  • Find the ESXi host running the VM
  • Use vSphere C# Client to connect to the ESXi host directly; Do not connect to the vCenter Server.
  • Locate the VM in the vSphere C# Client
  • Right-click on the VM, Guest, End VMware Tools Install

vmware.tools.upgrade.in.progress.02

  • VMware Tools status changes back to running.

vmware.tools.upgrade.in.progress.03

Then I can vMotion the VM or run the VMware Tools installation again.

Fix App Stuck on Updating in IOS

The Google Keep app on my iPhone 5s with IOS 9.3.3 is stuck on updating. I could neither launch the app nor delete it normally by holding down the icon (the X icon showed up, but nothing happened when clicking it). I tried the following fixes on the web, but no luck.

  • cancel the update and retry updating
  • reboot the phone and delete the app or retry updating
  • log out the App store on the phone (Settings, iTunes & App Store, Apple ID, Sign Out) and log in and retry updating

Some other suggestions are to reset the phone or connect the iPhone to a computer with iTune and delete the stuck app from iTune. I don’t want to either of these.

Luckily I found the following solution to delete the stuck app and re-download it from the App store.

  • Navigate to Settings, General, Storage & iCloud Usage, Manage Storage
  • Find the stuck app and click on it
  • Click Delete App
  • Then go back to App Store and re-download it under Updates, Purchased

vCSA “Syslog endpoint servername:514 is unreachable” Error

I am configuring the vCSA syslog to a third-party syslog server (e.g. a Splunk forwarder) via UDP port 514 (see the instruction in http://www.virtuallyghetto.com/2015/03/a-preview-of-native-syslog-support-in-vcsa-6-0.html). The syslog server receives the log from the vCSA. However, the VMware Syslog Service Health Messages reports a “Syslog endpoint servername:514 is unreachable” critical error.

It turns out the vCSA syslog uses the TCP port 514 for the syslog server health check. Since my syslog server (like many normal syslog servers) only licenses on the UPD port 514, the vCSA health check reports the syslog sever is not reachable.

Solution

  • Find a TCP port that the syslog server is licensing. Any licensing TCP port should work, it does not have to relate to the syslog.
  • SSH to vCSA
  • cd /etc/vmware-syslog
  • vi vmware-syslog-health.properties
  • Change the “cls.strata.ping.port” setting to the TCP port licensing on the syslog server (the default is 514)
  • Save the setting
  • Restart the VMware Syslog Service
  • Check the VMware Syslog Service Health Messages, it should show “Syslog endpoint <servername>:<tcp port> reachable”

One More Reason Not to Disable IPv6

Almost every existing operating system supports IPv6 and enables it by default nowadays. Some system admins still like to disable IPv6, because they think they would not deploy IPv6 in the near future. However, disabling IPv6 can be against the software vendor recommendation or experience the unexpected bug.

For example, Microsoft do not recommend disabling IPv6 in Windows. See “IPv6 for Microsoft Windows FAQ” and “How to disable IPv6 or its components in Windows”.

Recently, VMware ESXi 6.0.x has a known issue when IPv6 is disabled. See “Provisioning the TCP/IP stack does not work when IPv6 support is disabled on the host (2146023)

To avoid the unexpected issue, we should  leave IPv6 enabled (the default).

Use WinSCP to Transfer Files in vCSA 6.7

This is a quick update on my previous post “ Use WinSCP to Transfer Files in vCSA 6.5 ”. When I try the same SFTP server setting in vCSA 6.7...