Configuring VCSA 6.5 Backup Lessons Learned

vCenter Server Appliance (vCSA) 6.5 comes with the built in backup functionality. Starting a backup is quite easy - login the vCSA web console and click Backup button on the Summary page (see this post for the step-by-step screen shots).
Even it looks a very simple task, I have learned a few lessons when configuring the vCSA backup.
Lesson #1: vCSA backup location is <host_name>/<folder_name>
If using FTP protocol, the backup location is not just the FTP server host name or IP address; it MUST include the folder name. There is a “/” between the host name and folder name.
Otherwise, the error message is “FTP location is invalid”.

Lesson #2: vCSA backup supports the FTP virtual host name if entering the username correctly - <ftp virtual hostname>|<ftp username>
See my Lesson #2 in “Setting Up IIS 8 FTP Server Lessons Learned” about the FTP virtual host name login. There is a “|” between the hostname and username.
Otherwise, the error message is “Access to the remote server is denied. Check your credentials and permissions”.

Lesson #3: Use curl to troubleshoot vCSA backup error
After entering the correct settings, vCSA backup wizard validates the settings and starts the backup. The backup fails with “BackupManager encountered an exception. Please check logs for details”, but it does not provide much details or the location of the log file.

After some digging, I found the backup log file in /var/log/vmware/applmgmt/backup.log. In the log file, there is a curl error “Connection time-out”.

This gives me a hint that vCSA backup uses curl to transfer the backup file from vCSA to the FTP location. Recently I am also learning curl to transfer file, so I’m a little familiar with curl. (I will publish what I learn from curl in a future post).
From vCSA console, enter “curl -u <ftp user>:<password> -l <ftp server>”. It should list the file and directory on the FTP server. But I got the timeout error. I also tried running curl on a Windows computer, and got the timeout error too. This leads to me think the problem is on the FTP server. Finally the fix is to restart the FTP service (see Lesson #1 on “Setting Up IIS 8 FTP Server Lessons Learned”).
I am not sure why the wizard was able to successfully validate the FTP server setting when the FTP server connection is blocked by the Windows Firewall. When troubleshooting the Windows Firewall, I thought I could use the FTP command to connect to the FTP site, but using curl would fail. I’m not 100% sure about this, since I can’t replicate the issue again. After restarting the Microsoft FTP service, everything is working okay.
Anyway, using curl is the best tool to troubleshoot the vCSA backup failure.
Lesson #4: vCSA backup location must be an empty folder
After successfully running a backup, I try running the backup one more time with the same setting. I got the following error. (PS. In the screenshot below, I removed the virtual hostname on the FTP site, so I can just use the username).

Setting Up IIS 8 FTP Server Lessons Learned

To test vCSA 6.5 built-in backup, I need a FTP server. Since I already have a Windows Server 2012 R2 running IIS 8 with web service, adding the FTP server feature is just a few clicks.

Even I have not used the Microsoft FTP server since IIS 6, and there are lot of changes between IIS 6 and IIS 8, I thought setting up the FTP server should be a piece of cake. I was wrong! The following are what I have learned on setting up the FTP server in IIS 8.

Lesson #1: Windows Firewall

After installing the FTP service and creating a new FTP site in IIS Manager, I can’t connect to the FTP site from a remote computer; FTP from the server to itself is okay. It must be a Windows firewall issue.

• I check the Windows Firewall’s Inbound Rules, three FTP rules are created and enabled; and Outbound Rules, two FTP rules are created and enabled. I guess they are automatically created by the FTP service installation. These rules look right, but I still can’t connect from a remote computer.

• Disabling the Windows Firewall on the server, I can connect. This confirms the Windows Firewall causing the issue, but what is the problem? I don’t want to disable Windows Firewall.
• The default FTP rules are allowed the program “%windir%\system32\svchost.exe”. I’m not sure what the executable runs the FTP service. (Later, I find it via Microsoft FTP Service, General, Path to executable: “C:\Windows\system32\svchost.exe -k ftpsvc”)
• I created my own FTP rules required in my case - two inbound rules and one outbound rule (highlighted in above pictures) with the same protocol and port number, except that I allow any program. This works! I can connect to the FTP site from a remote computer. (Actually, see Lesson #2 below - it’s not fully working yet. I get another error after entering the login name).
• I think the default FTP rules don’t work, until I find this post.
• I delete the FTP rules I created, and restart the “"Microsoft FTP Service”. The FTP connect is still working.

Summary:

• When troubleshooting issues related to Windows Firewall, restart the application service or the server after adding or changing the rules.
• Restarting the FTP site in IIS Manage does not work; disabling and enabling the firewall or rule does not work. Restarting the FTP service is required.

Lesson #2: FTP site virtual host name

After the connection problem is resolved (see lesson #1), I continue further on the FTP login. However, after entering the user name, I get the error message “530 Valid hostname is expected. Login failed”.

After searching the error message, I learn about the FTP virtual host name

• Discussion on https://forums.iis.net/t/1161830.aspx, scroll down to the comment by Robert McMurray
• Microsoft official publish https://www.iis.net/learn/publish/using-the-ftp-service/using-ftp-virtual-host-names-in-iis-7

In the past I had used the IIS web site virtual hostname to handle multiple web sites on a single IP address and port number. But I don’t recall if the FTP service in IIS 6 has the host name option. When creating the FTP site, I entered the DNS name of the FTP site as the host name.

Summary:

• use <ftp virtual hostname>|<ftp username> as the login name for the FTP server uses the virtual hostname
• 
• If you are not going to run multiple FTP sites on the same IP address and port number, leave the host name blank.

VUM 6.5 “Cannot download patch definitions” via UMDS 6.5 Work Around

My nested vSphere lab environment does not have the access to the Internet (there is no physical network adapter as the uplink on the lab port group). To update and patch the ESXi host using vSphere Update Manager (VUM), I installed Update Manager Download Service (UMDS) on a Windows Server VM with dual NICs - one for Internet, another for the lab port group. Use the UMDS to download the update, configure IIS as the web server for the update repository, and configure VUM to use the http share repository. It worked fine in vSphere 6.0 and 6.2.

Recently I upgraded the lab environment to vSphere 6.5. The vCenter Server Appliance and ESXi hosts are upgraded to 6.5 successfully.

vCenter Server Appliance 6.5 bundles with VUM. VUM no longer requires a Windows Server. For UMDS, it can be installed on a Windows or Linux server. Since I already have a Windows server for UMDS. I continue using it instead of setting up a Linux server. But UMDS can’t be upgraded from the previous version to 6.5.

I uninstalled UMDS 6.0 and SQL Server 2012 Express on the server, and installed UMDS 6.5 with SQL Server 2012 Express from the ISO. I used the UMDS 6.0 repository folder for UMDS 6.5 and configured UMDS to download the host update only. Since IIS is already set up and I used the same repository folder, no change is needed in IIS. UMDS 6.5 successfully downloaded the update files from VMware.

I configured the VUM 6.5 to use the IIS shared repository. VUM validated the URL successfully. However, when I clicked “Download Now” button in VUM, I got the “Cannot download patch definitions” error.

Troubleshooting

• Verify the UMDS URL is accessible
  • enter http://umds-server/umds/version.txt in a web browser
• Look for ERROR in the VUM log file
  • SSH to vCSA
  • cd /var/log/vmware/vmware-updatemgr/vum-server
  • grep ERROR vmware-vum-server-log4cpp.log
  • found the following error in the log
  • Failed to download metadata zip file: http://umds-server/umds/hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip
• Check the repository folder in the UMDS server, there is no vmw-ESXi-6.5.0-metadata.zip file under hostupdate\vmw

Solution

• A similar issue happened in vSphere 5.5 (KB2061622)
• I can’t find a patch available for ESXi 6.5 yet, but I can apply the similar work around by removing <metadata> reference for vmw-ESXi-6.5.0-metadata.zip in
  hostupdate\vmw\__hostupdate20-consolidated-metadata-index__.xml
• Now the download patch definition is successful when I click “Download Now” button in VUM
• However, running UMDS again will re-add the vmw-ESXi-6.5.0-metadata.zip reference, and VUM will fail to download the patch definition
• This may be a bug in vSphere 6.5 (like vSphere 5.5). Hope VMware can fix in the next update

Stop A Task Stuck in vCenter Server Appliance

1. find out the name of vSphere host running the stuck task if possible
2. SSH to the vCenter Server server appliance
3. service vmware-vpxd restart
• After restart the vmware-vpxd service on the vCSA, the stuck task should disappear from the vSphere web client
• However, the task may be still running on the vSphere host
• Use the following steps to stop the stuck task on the vSphere host
4. SSH to the vSphere host running the task
5. /etc/init.d/hostd restart
6. /etc/init.d/vpxa restart

VSAN 6.2 On-disk Format Upgrade Fails at 5%

I am working on upgrading our VSAN from 6.1 to 6.2. See this from the upgrade step overview.

After upgrading each VSAN host to ESXi 6.0U2 (the latest build 4510822 as of 11/01/2016), the last step is to upgrade the on-disk format from v2 to v3.

In our case, the on-disk format upgrade fails at 5% with the error message “General Virtual SAN error. Disk Format conversion failed due to unexpected error”.

However, check the disk format in VSAN cluster, Manage, Settings, Virtual SAN / Disk Management. A disk group is upgraded to the interim version 2.5 each time I run the on-disk format upgrade. In the screen shots below, I ran the on-disk format upgrade twice. Two of the disk groups are upgraded to v2.5.

I keep running the on-disk format upgrade. In our VSAN, we have 4 hosts with 2 disk groups on each node. The on-disk format failed six times. On the seventh time, all disk groups are upgraded to v2.5.

Then the upgrade moves forward to the next process - starting remove disks from one of the VSAN host.

I have not figured out the cause of the failure. Re-running the upgrade process until all the disk groups are upgraded to the format v2.5 is able to keep the process moving forward.

VMware Tools Stuck in “Upgrade in progress” Fix

I notice the VMware Tools status on some VMs (mostly Linux) is “Upgrade in progress”.

For these VMs, I cannot vMotion them to another host; and some of these VMs, the “Edit Settings” and “Edit Resource Settings” are grayed out.

Solution:

• Find the ESXi host running the VM
• Use vSphere C# Client to connect to the ESXi host directly; Do not connect to the vCenter Server.
• Locate the VM in the vSphere C# Client
• Right-click on the VM, Guest, End VMware Tools Install

• VMware Tools status changes back to running.

Then I can vMotion the VM or run the VMware Tools installation again.