vCenter Server 6.5 Native High Availability Feature Summary

• Available exclusively for vCenter Server Appliance (vCSA)
• Consist of three nodes – active, passive, and witness nodes
  • Passive and Witness nodes are cloned from the existing vCSA (active node)
• vCenter HA cluster can be enabled, disabled, or destroyed at any time
• There is a maintenance mode to prevent planned maintenance from causing an unwanted failover
• Use two types of replication between active and passive nodes
  • Native PostgreSQL synchronous replication for the vCenter Server database
  • A separated asynchronous file system replication for key data outside the database
• Two vCenter HA deployment workflows
  • Basic: all vCenter HA nodes are deployed within the same cluster
  • Advanced: the active, passive, and witness nodes are deployed to different clusters
• There is little benefit to using vCenter HA without also providing high availability at the Platform Service Controller layer
  • An external Platform Services Controller instance is required when there are multiple vCenter Server instances in an Enhanced Linked Mode configuration.
• Failover can occur when a host failure, or when certain key services fail
• For the initial release of vCenter HA, a recovery time objective (RTO) is about 5 minutes

I have already known about some of these information when testing vCenter HA in my lab. I highlighted the ones I learned from this white paper.

Source: “What’s New in VMware vSphere”" 6.5” technical white paper

New Year Resolution - Improve Productivity

Here is my another new year resolution in 2017 - improve productivity (see my previous 2017 new year resolution here. The source of these ideas are from http://www.businessinsider.com/bad-habits-that-killing-productivity-2016-12.

• Get out of the bed when the alarm clock buzzes
• Get enough sleep
• Do not keep the tablet next to the bed. I keep the smartphone next to the bed as my alarm clock
• Do not skip breakfast and drink some hot tea before going to the toilet in the morning
• Complete the hardest and most important tasks at the beginning of the day
• Do not check email throughout the day, especially in the middle of the night. When wake up in the morning, only check if there is missing call or text message. Do not read the email until later of the day
• Do not eat junk food or eat less junk food
• Focus on 3 ~ 5 of the most important goals and ignore the rest
• Do not sit all day and walk 50,000 steps in a week
• Do not multitask
• Do not skip the workout
• Do not look up the answer of a random question that just popped into your head. Write it down and search later
• Do not overplan the schedule, instead plan for 4 ~ 5 hours of read work each day
• Do not underplan
• Do not accept a meeting unless the person who requested it has put forth a clear agenda and stated exactly how much time they will need
• Abandon perfectionism

Lessons from Security Breaches

Here are my short summary of the article “Learning From A Year of Security Breaches” that are applicable to most of work environments.

• Centralize logs, including host, application, authentication, and infrastructure, into as few system as possible; make critical logs alertable; but be aware of user privacy in what you log
• You might not find the root cause of a beach because of weakness in the environment, systems or people; practicing incident response can indentify these weakness
• Attackers will target employee’s home, personal email, or device to breach the corporate security; Educate your employees to improve their security practices and involve the corporate security team even if they have personal security issues
• Avoid putting secrets and keys into source code
• Protect employees’ credential by integrating Single Sing On or Multi Factor Authentication
• Be aware of insider threats
• Measure and eliminate the security debt - cutting corners for fast growth

First Day Result of Improving Sleep Quality

Here is the first day result of following my 2017 new year resolution - improve sleep quality

• Went to bed at 10:2x p.m. According to my Fitbit, slept at 10:34 p.m. and woke up at 3:53 a.m., and time asleep 5 hours and 9 minutes with 10 minutes restless. I know I woke up at 2:19 a.m. and 3:24 a.m. to check the time on my Fitbit. Then I am fully awake from a dream at 3:24 a.m.
• Did not drink any sola the whole day
• Drank half glass of water before going to bed
• Did not exercise the whole day, and walked 7,578 steps
• Shut down the computer at 10:1x p.m., did not read on the phone and tablet in bed
• Kept the phone next to the bed as the alarm clock. When waking up, I checked if there is any phone call or text message (the anwers is no); I did not read any email even there are some on the phone.

Conclusion

• Going to bed at 10 p.m. may be too early for me. I may move to 10:30 p.m.
• When wake up, check for phone call or text message only. Do not read any email until 6 a.m. or the start of a normal day

New Year Resolution - Improve Sleep Quality

To improve the sleep quality in 2017, I have some plans (the source of these ideas from https://medium.com/personal-growth/how-to-wake-up-early-your-ultimate-blueprint-1f8bb2045b90)

• Sleep at least 6 hours and 30 minutes each day. I will go to bed at 10 p.m. except Saturday.
• Do not drink sola after 4 p.m. or 6 hours before bedtime. I do not drink alcohol and rarely drink coffee.
• Drink half glass of water before going to bed
• Do not exercise after 7 p.m. or 3 hours before bedtime
• Do not read or watch on the smartphone, tablet, or computer after 9 p.m. or 1 hour before bedtime. This one will be difficult.
• Do not put the smartphone or tablet next to the bed

Configuring VCSA 6.5 Backup Lessons Learned

vCenter Server Appliance (vCSA) 6.5 comes with the built in backup functionality. Starting a backup is quite easy - login the vCSA web console and click Backup button on the Summary page (see this post for the step-by-step screen shots).
Even it looks a very simple task, I have learned a few lessons when configuring the vCSA backup.
Lesson #1: vCSA backup location is <host_name>/<folder_name>
If using FTP protocol, the backup location is not just the FTP server host name or IP address; it MUST include the folder name. There is a “/” between the host name and folder name.
Otherwise, the error message is “FTP location is invalid”.

Lesson #2: vCSA backup supports the FTP virtual host name if entering the username correctly - <ftp virtual hostname>|<ftp username>
See my Lesson #2 in “Setting Up IIS 8 FTP Server Lessons Learned” about the FTP virtual host name login. There is a “|” between the hostname and username.
Otherwise, the error message is “Access to the remote server is denied. Check your credentials and permissions”.

Lesson #3: Use curl to troubleshoot vCSA backup error
After entering the correct settings, vCSA backup wizard validates the settings and starts the backup. The backup fails with “BackupManager encountered an exception. Please check logs for details”, but it does not provide much details or the location of the log file.

After some digging, I found the backup log file in /var/log/vmware/applmgmt/backup.log. In the log file, there is a curl error “Connection time-out”.

This gives me a hint that vCSA backup uses curl to transfer the backup file from vCSA to the FTP location. Recently I am also learning curl to transfer file, so I’m a little familiar with curl. (I will publish what I learn from curl in a future post).
From vCSA console, enter “curl -u <ftp user>:<password> -l <ftp server>”. It should list the file and directory on the FTP server. But I got the timeout error. I also tried running curl on a Windows computer, and got the timeout error too. This leads to me think the problem is on the FTP server. Finally the fix is to restart the FTP service (see Lesson #1 on “Setting Up IIS 8 FTP Server Lessons Learned”).
I am not sure why the wizard was able to successfully validate the FTP server setting when the FTP server connection is blocked by the Windows Firewall. When troubleshooting the Windows Firewall, I thought I could use the FTP command to connect to the FTP site, but using curl would fail. I’m not 100% sure about this, since I can’t replicate the issue again. After restarting the Microsoft FTP service, everything is working okay.
Anyway, using curl is the best tool to troubleshoot the vCSA backup failure.
Lesson #4: vCSA backup location must be an empty folder
After successfully running a backup, I try running the backup one more time with the same setting. I got the following error. (PS. In the screenshot below, I removed the virtual hostname on the FTP site, so I can just use the username).

Setting Up IIS 8 FTP Server Lessons Learned

To test vCSA 6.5 built-in backup, I need a FTP server. Since I already have a Windows Server 2012 R2 running IIS 8 with web service, adding the FTP server feature is just a few clicks.

Even I have not used the Microsoft FTP server since IIS 6, and there are lot of changes between IIS 6 and IIS 8, I thought setting up the FTP server should be a piece of cake. I was wrong! The following are what I have learned on setting up the FTP server in IIS 8.

Lesson #1: Windows Firewall

After installing the FTP service and creating a new FTP site in IIS Manager, I can’t connect to the FTP site from a remote computer; FTP from the server to itself is okay. It must be a Windows firewall issue.

• I check the Windows Firewall’s Inbound Rules, three FTP rules are created and enabled; and Outbound Rules, two FTP rules are created and enabled. I guess they are automatically created by the FTP service installation. These rules look right, but I still can’t connect from a remote computer.

• Disabling the Windows Firewall on the server, I can connect. This confirms the Windows Firewall causing the issue, but what is the problem? I don’t want to disable Windows Firewall.
• The default FTP rules are allowed the program “%windir%\system32\svchost.exe”. I’m not sure what the executable runs the FTP service. (Later, I find it via Microsoft FTP Service, General, Path to executable: “C:\Windows\system32\svchost.exe -k ftpsvc”)
• I created my own FTP rules required in my case - two inbound rules and one outbound rule (highlighted in above pictures) with the same protocol and port number, except that I allow any program. This works! I can connect to the FTP site from a remote computer. (Actually, see Lesson #2 below - it’s not fully working yet. I get another error after entering the login name).
• I think the default FTP rules don’t work, until I find this post.
• I delete the FTP rules I created, and restart the “"Microsoft FTP Service”. The FTP connect is still working.

Summary:

• When troubleshooting issues related to Windows Firewall, restart the application service or the server after adding or changing the rules.
• Restarting the FTP site in IIS Manage does not work; disabling and enabling the firewall or rule does not work. Restarting the FTP service is required.

Lesson #2: FTP site virtual host name

After the connection problem is resolved (see lesson #1), I continue further on the FTP login. However, after entering the user name, I get the error message “530 Valid hostname is expected. Login failed”.

After searching the error message, I learn about the FTP virtual host name

• Discussion on https://forums.iis.net/t/1161830.aspx, scroll down to the comment by Robert McMurray
• Microsoft official publish https://www.iis.net/learn/publish/using-the-ftp-service/using-ftp-virtual-host-names-in-iis-7

In the past I had used the IIS web site virtual hostname to handle multiple web sites on a single IP address and port number. But I don’t recall if the FTP service in IIS 6 has the host name option. When creating the FTP site, I entered the DNS name of the FTP site as the host name.

Summary:

• use <ftp virtual hostname>|<ftp username> as the login name for the FTP server uses the virtual hostname
• 
• If you are not going to run multiple FTP sites on the same IP address and port number, leave the host name blank.