vSphere Client 6.0 could not connect to vCenter server error “The client did not receive a complete response from the server”

Due to security concerns in the TLS v1.0, we start disabling it on the system. After disabling TLS v1.0 on a Windows machine, I got the following error when connecting to a vCenter Server 6.0 using the vSphere Client (legacy C# client).

Troubleshooting

• I am still able to login vCenter via the vSphere Web Client on the web browser
• Re-enabling TLS v1.0 on the machine, I am able to login vCenter via the vSphere Client again
• The vCenter server is a VCSA 6.0 Update 3e. TLS v1.1/v1.2 are enabled by default (KB2145796).

Solution

1. Download and install the latest vSphere Client 6.0. As of this writing, vSphere Client 6.0.0 build 6855219.
2. Navigate to the configuration file C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe.config
3. Edit the VpxClient.exe.config file by setting the parameters
   <add key = “EnableTLS12” value = “false” />
   as
   <add key = “EnableTLS12” value = “true” />

Additional notes

• An older build of the vSphere Client 6.0 may not have the “EnableTLS12” entry in the configuration file. Install the latest build from the link in step #1.
• Some Microsoft updates or .NET 4.6.2 may need to install on the machine. See KB2149000.

Set Visual Studio Code (VS Code) Default Language Mode When New File Opened

Visual Studio Code is my default PowerShell script editor. It is free and available in Windows, macOS, and Linux.

By default, it uses text as the script language when a new file is opened. To make it recognize the PowerShell cmdlet, I have to manually change the language mode (press F1, type Change Language Mode, then select PowerShell form the list of the language) or first save to the file to ps1. This becomes inconvenience overtime.

Luckily, a new setting “files.defaultLanguage” is added. I didn’t backtrack when this setting was added. It’s available in the current release 1.14.0 as of this writing.

To configure the default language setting,

• Open File, Preferences, Settings
• Add the following to set PowerShell as the default language mode
• “files.defaultLanguage": "powershell",
• Save the setting.json file

To set other supported languages as the default

• Delete “powershell” in the line
• Press Ctrl + Spacebar to select the option from the list

Use WinSCP to Transfer Files in vCSA 6.5

To use WinSCP to transfer files with vCSA, VMware KB2107727’s solution is temporarily changing the default shell from appliancesh to bash, then changing back after the transfer. This works in vCSA 6.0 and vCSA 6.5.

In vCSA 6.0, there is a tick to change the WinSCP’s Advanced, SFTP server setting to "shell /usr/lib64/ssh/sftp-server" (without the quotes) for file transfer without changing the default shell. See “Connect to vCSA using WinSCP”

VMware changed the OS from SLES to Photon in vCSA 6.5. The above setting doesn’t work anymore. There is no ssh directory under /usr/lib64/. However, a quick search and find the sftp-server is moved to /usr/libexec/. Using the setting “shell /usr/libexec/sftp-server” in vCSA 6.5 works fine.

Here is the detail instruction.

• Log in vCSA VAMI UI (https://vcsa-ip:5480)
• Under Access, enable SSH Login. (PS: enabling Bash Shell is not necessary)
• Open WinSCP, select File protocol: SFTP
• Enter the vCSA hostname, port number 22, root, and root’s password
• Click Advanced
• Under Environment, SFTP, Protocol options
• Set SFTP server to “shell /usr/libexec/sftp-server” (without the quotes)

PS: I tested the setting in WinSCP v5.9.6 build 7601 and vCSA v6.5.0.5600 build 4951144.

VCSA 6.5 “The appliance management service is not running” Fix

Scenario

In vSphere Web Client 6.5, under Home, Administration, Deployment/System Configuration, Nodes, the vCenter Server node shows an error message “The appliance management service is not running”. An error message “HTTP response with status code 503, 503 Service Unavailable (Failed to connect to endpoint: _serverNamespace = /vmonapi action =Allow _port = 8900" also appears in the web client.

Troubleshooting

• Login the VMware Appliance Management UI (https://psc:5480 or https://vc:5480). All the health status are good.
• SSH to VC appliance. Check service status (KB2109887)
• # service-control –list
• # service-control –status
• applmgmt (VMware Appliance Management Service) is running
• vmonapi (VMware Service Lifecycle Manager API) is not running

Solution

• Restart vmonapi service or restart all services
• # service-control --start vmonapi
• # service-control –start –all
• PS: if restarting all the services, it may take some time before all services turn back to Good (green) in the node’s Summary page. e.g. VMware Performance Charts service takes more than 30 minutes to change from Warning, Unknown, and then Good.

VCSA 6.5 Syslog vs vRLI’s vSphere Integration

I write this post after reading William Lam’s “What logs do I get when I enable syslog in VCSA 6.5?” and doing some of my experiment on my VCSA 6.5 and vRLI 4.5 setup.

Background

Recently I completed a fresh VCSA 6.5 (external PSC and VC) deployment with vRealize Operations Manager (vROPS) 6.6 and vRealize Log Insight 4.5 installation. In vROPS, I configured vSphere and vRLI solutions; in vRLI, I configured vSphere and vROPS integration. I thought I completed all the setup until reading William’s blog post.

Confusion

There are a lot of information on his blog post. I was a little lost at the beginning, and I was wondering: should I configure VCSA syslog to vRLI? Is the same as vRLI’s vSphere integration? If I read his blog carefully, I would find the answer there. I didn’t fully understand it until I did my own experiment. Here is the quote. I highlighted a few key points.

I personally think the vSphere Integration is a nice solution if you have both Windows vCenter Server and the VCSA and to be able to get data consistency between the two platforms from a logging standpoint. It is definitely useful if you need to quickly enable all ESXi hosts connected to the vCenter Server and have them remotely syslog to the vRLI instance. If you only have the VCSA, you would get more information by configuring the remote syslog capability in VCSA rather than using the vSphere integration feature of vRLI. This especially true if you need the vpxd.log which is generally required for troubleshooting and debugging vCenter Server issues when calling into VMware Support. The other added benefit to using the VCSA option is that structure log entries are processed directly on the VCSA rather than having to be remotely queried via the vSphere APIs, processed and then store in vRLI which would add additional load onto vRLI, especially if you need to configure additional vCenter Server instances.

Summary

I summarize based on my understanding of this topic here. Please refer his blog for the full details.

• VCSA 6.5 has a new remote syslog functionality comparing to VCSA 6.0. This function is not available in Windows vCenter Server 6.5
• VCSA 6.5’s remote syslog configuration is in the VAMI UI (https://[VCSA]:5480). This setting available in both PSC and VC for external deployment. See William’s post’s “Logs forwarded by VCSA Deployment Type” for the logs forwarded in different VCSA deployment type
• VCSA 6.0’s remote syslog configuration is in the vCenter via vSphere Web Client
• VCSA 6.5 has a new Enhanced Logging feature (see William’s blog for what the enhanced means; see my screen shots in this post for a better example)
• After completing vRLI’s vSphere integration, “enable streaming of events to syslog” is enabled (vSphere Web Client, vCenter, Configure, Advanced Settings, vpxd.event.syslog.enabled). This setting is mentioned in another person blog. I am not sure what the default VCSA setting is. Put it here for the reference only
• VCSA 6.5 remote syslog is not configured even completing vSphere integration in vRLI
• VCSA 6.5 remote syslog is “pushing” the logs to vRLI
• vRLI’s vSphere integration is “pulling” the logs from VCSA (via vSphere API). This supports both a Windows vCenter Server and VCSA.
• vRLI’s vSphere integration can also automatically configure the ESXi hosts connected to the vCenter Server and have them remotely syslog to vRLI. (vSphere Web Client, ESXi host, Configure, System/Advanced System Settings, Syslog.global.logHost)
• By default, vCenter Server log (vpxd.log) is not forwarded to a remote syslog server. It is recommended enabling it for troubleshooting purposes. (vSphere Web Client, vCenter, Configure, Advanced Settings, config.log.outputToSyslog; then restart vCenter Server service in System Configuration, Services, VMware vCenter Server)
• Other VCSA 6.5 logs can be forwarded to a remote syslog server. but it’s not supported by VMware. See the link at the end of William’s post for more details
• This is the most important and useful point I have learned. VCSA 6.5 remote syslog sends more information to vRLI comparing to vRLI’s integration. I think this is what the Enhanced Logging means. See my screen shots below. For example, I modified the Tools Upgrades option on a VM.
• Without VCSA remote syslog configured, vRLI has one entry in the log. It shows the name of the VM (highlighted in yellow)’s toolsUpgradePolicy is changed from “manual” to “upgradeAtPowerCycle”
• With VCSA remote syslog configured, vRLI has two entries in the log. In additional to the regular log, the second entry shows the name of the user made the change (highlighted in the red box).
• My recommendation is to configure both vRLI’s vSphere integration (for automate configuring the ESXi log host) and VCSA remote syslog (for the enhanced logging). This would duplicate some log entries in vRLI and consume more vRLI log storage. But it is well worthy!

vSphere 6.5 New Feature – VMware Orchestrated Restart

Let me back to the old ESXi 3 day – when I was just using the standalone ESXi hosts or vCenter without HA and DRS. In case of the power outage or air conditioning failure in the data center, all the ESXi hosts were powered down. Once the environment problem was resolved, I could manage the VM startup sequence by configuring the switched PDU to start the hosts accordingly, and configuring the VM startup order at the host level.

However, once I deployed vCenter Server with HA and DRS, I lost the control of the VM startup order. Because the VMs could be hosted at any host in the cluster. Someone said that I should not worry about the VM startup order in the cluster. Because the ESXi cluster would never go down if I had designed the infrastructure with enough redundancy. As we all know, we never have enough redundancy in a small ESXi deployment.

I have been curious why VMware do not “fix” this issue for so long. Until now, vSphere 6.5 introduces the VMware Orchestrated Restart feature. At the high level, the Orchestrated Restart, likes the VM affinity and anti-affinity rules, put the VMs in different VM groups and set the startup dependence among the VM groups. To learn more about this, please go to “What is VMware Orchestrated Restart?”.

I am so glad to know about this new vSphere 6.5 feature – one more reason to upgrading to vSphere 6.5.

vSAN Performance Service “Hosts Not Contributing Stats” Fix

I have a four-host vSAN cluster running vSAN 6.2. Recently the vSAN health’s Performance service check shows two of the hosts not contributing stats.

The following are all the steps that I tried during troubleshooting and ultimately fixing the issue in my environment. Some of the steps do not fix my issue, however they may be applicable to your situation. PS. I opened a VMware support case on this issue. The support engineer did not directly solve my issue. However, he did give the hint on the cause of the issue that led me to discover the solution.

1. Turn off and turn on the Performance Services in vSphere web client, vSAN cluster, Manage, Settings, Health and Performance.
2. Turn off the Performance Services, restart the vSAN management agent “/etc/init.d/vsanmgmtd restart”, then restart the service.
3. Place the vSAN host in the maintenance mode and restart the host.
4. SSH to the vCenter server appliance, restart the vmware-vpxd service “service vmware-vpxd restart”.
5. Verify the vSAN storage provider status of each vSAN host is online in vSphere web client, vCenter server, Manage, Storage Providers. If the host’s vSAN provider is offline, unregister the host’s storage provider and synchronize all vSAN storage providers. This brings the host’s vSAN storage provider back online.
   Caution: doing this can cause the VMs on the host to failover to other hosts in the cluster.
   
6. (I think this is to begin to lead me to the ultimate fix) Check the certificate info of each vSAN host in Storage Provider. They should be issued by the same Platform Service Controller (my vCenter is the vCSA wit the external PSC, instead of the embedded PSC). In my case, the certificate of the two “problem” vSAN hosts is issued by the VC host; the certificate of the “good” vSAN hosts is issued by the PSC host. I don’t know what the cause of these hosts having different certificate issuers, since I don’t have the history of how these PSC and VC were deployed.
   
   
7. To further confirm the ESXi host certificate is the problem
1. Login vCenter server as “administrator@vsphere.local’
2. Home, Administration, Deployment, System Configuration, Nodes, PSC node, Manage, Certificate Authority (if selecting VC node, there is no Certificate Authority tab under Manage)
3. Enter the password of “administrator@vsphere.local” again
4. Active Certificate, all the ESXi hosts are listed, except the two “problem” vSAN hosts
5. It makes sense why the certificates of the two “problem” vSAN hosts are missing here, because they are issued by the VC host, not the PSC host. But it does not make sense how they received the “problem” certificate since there is no Certificate Authority on the VC host.
   
8. Once the cause is identified, the fix is to re-issue the certificate to the two “problem” vSAN hosts.
9. In vSphere web client, the “problem” vSAN host, Manage, Settings, Certificate
1. Here is also showed the host certificate issuing by the wrong host (the VC host)
2. Click Renew to request a new certificate
3. Caution: Once clicking the Renew button, the host HA agent was restarted. Some VMs on the host failed over to the remaining hosts, even the VMs seem no downtime.
   Before renewing the certificate
   
   After renewing the certificate
   
10. Once the host certificates are re-issued by the PSC, the vSAN Performance service status is showed “Passed”
    

Conclusion

• The cause of the vSAN Performance service “Host Not Contributing Stats” in my case is the “problem” vSAN host having the wrong host certificate.
• I don’t know how these “problem” hosts received the wrong host certificate.
• When the vCSA with the external PSC, the host certificate is issued by the PSC host.
• Re-issuing or renewing the host certificate will restart the host HA agent. It can cause the VMs on the host migrating to other hosts.