Use WinSCP to Transfer Files in vCSA 6.7

This is a quick update on my previous post “Use WinSCP to Transfer Files in vCSA 6.5”. When I try the same SFTP server setting in vCSA 6.7, it doesn’t work. But it works when removing “shell” in the setting.

So for vCSA 6.7, the SFTP server setting in WinSCP is “/usr/libexec/sftp-server”

Since I upgraded my vCSA 6.5 to 6.7, I cannot test if the setting without “shell” woks in vCSA 6.5 or not. Please test and comment if you have vCSA 6.5 available.

09/15/2021 Update:

Since I discovered the above WinSCP setting in May 2020, I had not transferred files to vCSA using WinSCP until today. But the above setting doesn't work anymore. My vCSA had been updated multiple times for 6.7 patches. 

My current vCSA version 6.7 Update 3n (6.7.0.48000). The WinSCP SFTP server setting needs to be "shell /usr/libexec/sftp-server", like vCSA 6.5.

So if you have need to transfer files to vCSA 6.7 via WinSCP, try one of the following settings

• shell /usr/libexec/sftp-server
• /usr/libexec/sftp-server

Nagios Core and NagiosQL Installation on CentOS 7

This post is to document my Nagios Core 4.4.6 and NagiosQL 3.4.1 installation on CentOS 7. They are the latest available version at the time of this writing (May 2020).

The main reason of this post is I cannot find the updated instruction to install the latest version of NagiosQL on CentOS 7. When I follow the instruction on the outdated blog posts, I run into some issues. For example, the official CentOS 7 repository only has PHP 5.4, but NagiosQL 3.4.1 requires PHP 5.5.0 or later; and MySQL is no longer in CentOS repository. Another reason is I want to install the latest version packages (e.g. PHP 7.4), instead of the older version.

Nagios Core and Nagios Plugins Installation on CentOS 7

I follow the instruction on Nagios Support Knowledgebase without any major issue. The only modification is to get the latest version of Nagios Core 4.4.6, instead of 4.4.5; and the latest version of the Nagios Plugin 2.3.3, instead of 2.2.1.

NagiosQL Installation on CentOS 7

1. Install PHP 7.4 from Remi and EPEL repositories

As I mention earlier, the official CentOS 7 repository only has PHP 5.4. This doesn’t meet the NagiosQL 3.4.1 requirement. You can check the installed PHP version by php -v.

yum install epel-release
yum install http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
yum install yum-utils
### install PHP 7.4
yum-config-manager --enable remi-php74
yum install php php-mcrypt php-cli php-gd php-curl php-mysql php-ldap php-zip php-fileinfo
### verify PHP 7.4 is installed
php -v

2. Install MySQL from the community repository

I follow the instruction on this post without any issue.

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
yum install mysql-server
systemctl start mysqld
### change MySQL root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases
mysql_secure_installation

3. Install NagiosQL required packages

yum install libssh2 libssh2-devel mysql php-mysql php-pear php-devel
### install ssh2.so version 1.2 beta that support PHP 7.4
pecl install ssh2-1.2
### add extension=ssh2.so to /etc/php.ini under Dynamic Extensions
vi /etc/php.ini

4. Download and extract NagiosQL 3.4.1 archive in Nagios document root (/usr/local/nagios/share)

cd /usr/local/nagios/share
curl -L -O https://downloads.sourceforge.net/project/nagiosql/nagiosql/NagiosQL%203.4.1/nagiosql-3.4.1-git2020-01-19.tar.gz
tar xzf nagiosql-3.4.1-git2020-01-19.tar
mv nagiosql-3.4.1 webadmin
chown -R nagios:nagios webadmin/

### create the NagiosQL configuration directory
mkdir /usr/local/nagios/nagiosql
chown -R apache:apache /usr/local/nagios/nagiosql/

5. Set up PHP Timezone and restart Apache web server

See here to get the list of timezone

### set date.timezone = 'America/Los_Angeles'
vi /etc/php.ini

systemctl restart httpd

6. Start NagiosQL web installer

• Open the URL in a browser: http://nagiosserver/nagios/webadmin/install/index.php
• Click Start Installation
• Verify the system meets all the requirements. Here is where I found out the PHP 5.4 in CentOS 7 doesn’t meet the requirement
• Click Next
• On NagiosQL Installation: Setup page
  • Enter NagiosQL DB password, root password (the root password is MySQL root password), and NagiosQL admin password
  • Check the checkboxes
    • “Drop database if already exists?”
    • “Import Nagios sample config?” (optional)
    • “Create NagiosQL config paths?”
  • set NagiosQL config path: /usr/local/nagios/nagiosql
  • set Nagios config path: /usr/local/nagios/etc
• Click Next
• On NagiosQL Installation: Finishing Setup page, it should be all green if everything is right
• Delete the NagiosQL install directory

7. Access NagiosQL web UI

• Open the URL in a browser: http://nagiosserver/nagios/webadmin
• login with NagiosQL admin and password

8. Integrate NagiosQL with Nagios

• Navigate to Administration -> Administration -> Config targets
• Click Modify icon next to localhost
• On Configuration domain administration page
  • Configuration directories section should be all set. No change is needed
  • Nagios configuration files and directories section, verify the following settings
    • Nagios base directory: /usr/local/nagios/etc/
    • Import directory: /usr/local/nagios/etc/objects/
    • Picture base directory: (blank)
    • Nagios command file: /usr/local/nagios/var/rw/nagios.cmd
    • Nagios binary file: /usr/local/nagios/bin/nagios
    • Nagios process file: /run/nagios.lock
    • Nagios config file: /usr/local/nagios/etc/nagios.cfg
    • Nagios cgi file: /usr/local/nagios/etc/cgi.cfg
    • Nagios resource file: /usr/local/nagios/etc/resource.cfg
  • Select 4.x in Nagios version
  • Leave Access group “Unrestricted access”
  • Check Active checkbox
  • Click Save
• Edit Nagios Core configuration file
  • Edit Nagios configuration file /usr/local/nagios/etc/nagios.cfg
  • Comment all cfg_file and cfg_dir entries
  • Add the following cfg_file and cfg_dir entries

  cfg_file=/usr/local/nagios/nagiosql/commands.cfg
  cfg_file=/usr/local/nagios/nagiosql/contactgroups.cfg
  cfg_file=/usr/local/nagios/nagiosql/contacts.cfg
  cfg_file=/usr/local/nagios/nagiosql/contacttemplates.cfg
  cfg_file=/usr/local/nagios/nagiosql/hostdependencies.cfg
  cfg_file=/usr/local/nagios/nagiosql/hostescalations.cfg
  cfg_file=/usr/local/nagios/nagiosql/hostextinfo.cfg
  cfg_file=/usr/local/nagios/nagiosql/hostgroups.cfg
  cfg_file=/usr/local/nagios/nagiosql/hosttemplates.cfg
  cfg_file=/usr/local/nagios/nagiosql/servicedependencies.cfg
  cfg_file=/usr/local/nagios/nagiosql/serviceescalations.cfg
  cfg_file=/usr/local/nagios/nagiosql/serviceextinfo.cfg
  cfg_file=/usr/local/nagios/nagiosql/servicegroups.cfg
  cfg_file=/usr/local/nagios/nagiosql/servicetemplates.cfg
  cfg_file=/usr/local/nagios/nagiosql/timeperiods.cfg
  
  cfg_dir=/usr/local/nagios/nagiosql/hosts
  cfg_dir=/usr/local/nagios/nagiosql/services

9. Verify Nagios Core config files

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

When I run the above command, I get the error message of missing command.cfg file, etc. I go back to NagiosQL web UI, in each main section Supervision, Alerting, Commands, and Specialties, click “Write config file” to generate these files. Then the command reports no error or warning.

10. Restart Nagios Core service

systemctl restart nagios

Now Nagios Core and NagiosQL are successfully set up. You can view the monitoring status in the Nagios web UI and modify the monitoring via NagiosQL web UI.

Three Ways to Split String in PowerShell

This post is to summarize three ways to split a string in PowerShell.

In the following example
$t = “1000ABC2000abc"
[regex]$rx1 = “\d+”
[regex]$rx2 = “ABC"

• Using a regex object’s split method
  • In this method, the regex works and the match is case-sensitive
    
    PS > $rx1.split($t)
    
    

    ABC
    abc

    PS > $rx2.split($t)
    1000
    2000abc

    
    
• Using a string object’s split method
  
  • In this method, the regex does not work and the match is case-sensitive
    
    PS > $t.split($rx1)
    1000ABC2000abc
    
    

    PS > $t.split($rx2)
    1000
    2000abc

    
    
• Using a string object’s split operator
  • In this method, the regex works and the match is case-insensitive
    
    PS > $t -split $rx1
    
    

    ABC
    abc

    PS > $t -split $rx2
    1000
    2000

Enable HTTPS with Let’s Encrypt SSL certificate on Pi-hole web interface

By default, the Pi-hole web interface runs on HTTP, including the web admin panel URL. I want to secure the Pi-hole password with HTTPS when entering in the web browser. To enable HTTPS, I need a SSL certificate. My options are creating a self-signed SSL certificate, buying a retail SSL certificate from a public CA, or using Let’s Encrypt free SSL certificate.

For the home setup, the Let’s Encrypt SSL certificate is a perfect fit. The certificate works all the major web browsers, so no security warning in the browser; and it’s free. Just need to renew it every 90 days.

Before proceeding the following instruction, make sure you meet these two perquisites.
1. You own a public domain name.
2. You have the access to modify the public DSN setting of your domain name. The instruction on how to do this varies from the DNS hosting vendor. Please consult with your DNS hosting vendor for the detail.

1. Issue Let’s Encrypt SSL certificate

Let’s Encrypt recommends the Certbot ACME client to automate the issuance and installation. Because I don’t want my Pi-hole web interface accessible on the internet. I have to run Certbot with the manual option to issue the certificate. Here is how I do that.

• Login my pi-hole via SSH
• Install Certbot
  • $ sudo apt install certbot
• Run certbot to issue a certificate for Pi-hole FQDN. In my example, the FQND is pihole.sfitpro.com.
  • $ sudo certbot certonly --manual --preferred-challenges dns --cert-name pihole.sfitpro.com -d pihole.sfitpro.com
• During this process, certbot will prompt to add a DNS TXT record “_acme-challenge.pihole.sfitpro.com” with the value created by certbot.
• Login my domain name DNS hosting site and add a “_acme-challenge.pihole.sfitpro.com” TXT record with the provided value to verify my ownership of the domain name.
• After the TXT record is added, continue with certbot to finish the certificate issuance.
• The private key and issued certificate are saved in /etc/letsencrypt/live/.

2. Enable HTTPS on Pi-hole web interface

• Create a file called combined.pem in the Let’s Encrypt certificate directory
  • $ sudo su
  • $ cd /etc/letsencrypt/live/pihole.sfitpro.com/
  • $ cat privacy.pem cert.perm | tee combined.pem
• Ensure the lighttpd user (www-data) can read the certificates
  • $ sudo chown www-data -R /etc/letencrypt/live
• Create a file called external.conf in /etc/lighttpd/ with the following content

$HTTP["host"] == "pihole.sfitpro.com" {
# Ensure the Pi-hole Block Page knows that this is not a blocked domain
setenv.add-environment = ("fqdn" => "true")

# Enable the SSL engine with a LE cert, only for this specific host
$SERVER["socket"] == ":443" {
ssl.engine = "enable"
ssl.pemfile = "/etc/letsencrypt/live/pihole.sfitpro.com/combined.pem"
ssl.ca-file = "/etc/letsencrypt/live/pihole.sfitpro.com/fullchain.pem"
ssl.honor-cipher-order = "enable"
ssl.cipher-list = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"
ssl.use-sslv2 = "disable"
ssl.use-sslv3 = "disable"
}

# Redirect HTTP to HTTPS
$HTTP["scheme"] == "http" {
$HTTP["host"] =~ ".*" {
url.redirect = (".*" => "https://%0$0")
}
}
}

• Restart the lighttpd service
  • $ sudo systemctl restart lighttpd

3. Add the Pi-hole FQDN on Pi-hole

• Create a file called lan.list in /etc/pihole/
• Add the following to the file
  • <pi-hole-ip-address> <pi-hole-fqdn>
  • e.g. 192.168.1.19 pihole.sfitpro.com
• Create a second dnsmasq config file called 02-lan.conf in /etc/dnsmasq.d/
• Add the following to the file to reference the lan.list file created above
  • addn-hosts=/etc/pihole/lan.list
• Restart the DNS service on pi-hole$ sudo pihole restartdns
  • $ sudo pihole restartdns

Now when entering the Pi-hole FQDN in the browser, it will be redirected to the HTTPS page with a valid SSL certificate.

Match a string ending with a dollar sign ($) and containing a variable in PowerShell

In my previous post, I can match a string ending with a dollar sign ($) using the single quote with the expression ‘\$$’. Because the single quote protects the PowerShell automatic variable $$ from being evaluated. But it brings up my next question, how about the same expression also includes other variable that should be evaluated. Like this example.

PS C:\Temp> $name = 'smith'
PS C:\Temp>
PS C:\Temp> 'contoso\john.smith$' -match "$name"
True
PS C:\Temp> 'contoso\john.smith$' -match '$name'
False

Obviously, I have to use the double quote to evaluate the variable $name before sending the expression to the regex engine. But I also need the single quote for the variable $$.

PS C:\Temp> 'contoso\john.smith$' -match '$name\$$'
False
PS C:\Temp> 'contoso\john.smith$' -match "$name\$$"
False

Here is my solution - using double quote with both the regex escape character backslash(\) and PowerShell escape character backpack(`).

PS C:\Temp> 'contoso\john.smith$' -match "$name\$`$"
True

Let me explain

• Use double quote to evaluate the variable $name
• Use `$ to prevent the automatic variable $$. Using the PowerShell escape character (`) because it’s the PowerShell evaluation, not regex evaluation. So do not use \$ on the second $.
• After the variable evaluation, the expression becomes smith\$$. This is passed to the regex engine. As expected, this regex matches the string ending with smith$.

Match a string ending with a dollar sign ($) in PowerShell

I want to match a string ending with a dollar sign ($) (e.g. ‘contoso\john.smith$’) in PowerShell. Using the regular expression (regex) should be simple, like \$$.

However, I run into some problem in PowerShell. I posted my question in Reddit for help. With the comments from the community, I think I finally understand how to handle this issue. This blog post is to summarize my understanding.

• -match operator uses the regular expression syntax.
• The scape character in regex is the backslash(\). Normally the regular PowerShell escape character, the backtick(`), should not use in the regex expression. See my next post on using both scape characters (\ and `) in one expression.
• To match a string ending with a dollar sign ($), the regex should be \$$. The first $ is for the literal $, so it is escaped by \. The second $ is an anchor which matches the end of a string, so it is not escaped by \.
• However $$ is an automatic variable in PowerShell.
• When the expression(\$$) is doubled-quoted, PowerShell evaluates the variable $$ first before sending to the regex engine. “\$$” becomes “\{value of $$}” when being parsed by the regex engine. So it returns False.

PS C:\Temp> 'contoso\john.smith$' -match "\$$"
False

• When the expression(\$$) is single-quoted, PowerShell does not evaluate any variable. The expression \$$ is parsed by the regex engine. So it returns True.

PS C:\Temp> 'contoso\john.smith$' -match '\$$'
True

In the case, I can match a string ending with $ by using the single quote with the expression '\$$'. However, this will not work if the expression includes other variable that should be evaluated. I will post my solution in the next post.

My Pi-hole blocklists

After setting up the Pi-hole on the Raspberry Pi, I start looking for additional blocklists to block as much as unnecessary traffic on my home network. 

Searching on the Pi-hole Userspace, I found many discussions about the blocklists. I want to share my blocklists on this post, so you can save time.

My goals are to use the blocklists that

• Are from the reliable source
• Are updated regularly
• Provide additional filter categories that are safe for family
• Are free of charge

As mentioned in my “Select Upstream DNS server for Pi-hole” post, I may be able to use OpenDNS or CleanBrowsing DNS as my upstream DNS server to achieve the same goals without using the blocklists. But these blocklists become necessary after I set up DoH with Cloudflare DNS because Cloudflare DNS doesn’t provide content filtering.

The following are the blocklists in my Pi-hole. You can copy and paste them under Pi-hole’s admin page, Settings, Blocklists, and click “Save and Update”.

At the time of this writing, these lists block over 3,900,000 unique domains. I have to admit some legit domains also being blocked, e.g. the Target.com's Weekly Ad. But they can be fairly easy to identify and add to the whitelist. I will share in a future post.

Please provide your feedback if you have a better list or issue with my list. I will keep the list up-to-date.

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://blocklist.site/app/dl/ads
https://blocklist.site/app/dl/crypto
https://blocklist.site/app/dl/drugs
https://blocklist.site/app/dl/fraud
https://blocklist.site/app/dl/fakenews
https://blocklist.site/app/dl/gambling
https://blocklist.site/app/dl/malware
https://blocklist.site/app/dl/phishing
https://blocklist.site/app/dl/piracy
https://blocklist.site/app/dl/porn
https://blocklist.site/app/dl/proxy
https://blocklist.site/app/dl/ransomware
https://blocklist.site/app/dl/redirect
https://blocklist.site/app/dl/scam
https://blocklist.site/app/dl/spam
https://blocklist.site/app/dl/tracking
https://dbl.oisd.nl/

Set up Cloudfared DoH for Pi-hole

I would prefer using DNS over HTTPS (DoH) to increase privacy and security. Mozilla Firefox is the first web browser implementing DoH, and many other browsers follow. But how can I utilize DoH for the non-browser DNS request or enable DoH on each device that connects to my home network?

Pi-hole has a document to configure DNS-Over-HTTPS. To automate the install and configure Cloudfared on a Raspberry Pi running Raspbian, I create a bash script. I also add the steps to lock down the cloudfared account.

Here are how set up Cloudfared DoH using the script.

• Make sure the Pi-hole is set up on your Raspberry Pi. See my post “Set up Pi-hole on a Raspberry Pi”.
• Download the script from my Github on your Raspberry Pi
  • curl -O https://raw.githubusercontent.com/sfitpro/pi-hole/master/setup.cloudflared.doh.for.pi-hole.sh
• Grant the execution permission to the script
  • chmod +x setup.cloudflared.doh.for.pi-hole.sh
• Run the script
  • sudo ./setup.cloudflared.doh.for.pi-hole.sh
• Configure the Pi-hole to use custom Upstream DNS server 127.0.0.1#5053

If all the setup is correct, you will notice the DNS queries on Pi-hole are answered by localhost.

Select Upstream DNS server for Pi-hole

Once the Pi-hole is up and running, the first thing I want to configure is its upstream DNS servers. The upstream DNS servers can provide additional filters (e.g. adult-related sites, social networking sites, etc) that are not included in the Pi-hole default installation.

I have been using OpenDNS as my home router's upstream DNS servers for a long time. I sign up an OpenDNS account so I can customize the kind of content to block or the sites to be whitelisted. It works great.

On the Pi-hole admin page, under Settings —> DNS. It has a list of built-in upstream DNS servers, e.g. Google, OpenDNS, Quad9, Cloudflare, etc.

• OpenDNS: OpenDNS is my original first choice since I have been using its filter for a long time as I mentioned above. My only concern with OpenDNS is that Cisco acquired OpenDNS in 2015; and not sure whether this free service will continue to be maintained. But I don’t have other good options at the time.
• Google DNS: Google DNS has a relatively fast response time. It doesn’t provide content filtering. Google’s ad tracking business makes me stay away from its products when I can. So I stop using Google DNS.
• Cloudflare DNS: Cloudflare DNS has a fast response time. It doesn’t provide content filtering. After knowing Mozilla includes Cloudflare DoH in their newer version of Firefox, I trust Cloudflare’s privacy practice. Combining with Pi-hole other filtering feature, I switch to Cloudflare DNS as my upstream DNS servers until I learn about CleanBrowsing DNS.
• CleanBrowsing DNS: CleanBrowsing DNS’s free filters (Security, Adult, and Family) are silmiar to OpenDNS, except the free plan doesn’t offer custom filters (e.g. gaming, grambling, etc). However, its free adult or family filter can filter the search engines (Google and Bing) result in safe mode, which I don’t see in other DNS filter. So CleanBrowsing DNS’s audlt filter is my choice for the Pi-hole upstream DNS at the home network. (PS. the family filter blocks Reddit site. I agree some contents on Reddit is not appropriate, but it also has many good technical disucssion.)

CleanBrowsing DNS setup on Pi-hole

• http://<pihole.ip.address>/admin
• Setting, DNS, Upstream DNS Servers
• Custom 1 (IPv4): 185.228.168.10
• Custom 2 (IPv4): 185.228.169.11

Set up Pi-hole on a Raspberry Pi

My first and main usage for a Raspberry Pi is to set up Pi-hole to block Ad for all devices on my home network.

Installation

The Pi-hole instalaltion is simple.

curl -sSL https://install.pi-hole.net | bash

Basic operation command

• check status
  • sudo pihole status
• check version
  • sudo pihole -v
• update Pi-hole
  • sudo pihole -up
• change Pi-hole admin password
  • sudo pihole -a -p
• update Pi-hole filter list (gravity.list)
  • sudo pihole -g
• restart Pi-hole
  • sudo pihole restartdns
• list domains in whilelist
  • sudo pihole -w -l
• add a domain to whitelist
  • sudo pihole -w <domain>
• remove a domain from whitelist
  • sudo pihole -w -d <domain>
• list domains in blacklist
  • sudo pihole -b -l

Once the pi-hole is up and running, we can change the the DNS setting of the router’s DHCP to the pi-hole IP address, or manaully set it to the devices with the static IP address.

Pi-hole comes the default ad block list. I will discuss the selection of Upstream DNS servers and add additional filter in the next post.

Set up a Raspberry Pi

I’m going to write a series of posts about Raspberry Pi and how I use it to secure and optimize my home network. This first post covers the basic setup procedure. Please refer to the official setup guide if you need the step-by-step instruction.

Hardware

• Raspberry Pi 3 Model B. At the time of writing, the latest board is Raspberry Pi 4 Mode B, which is more powerful. But any version Raspberry Pi board should work.
• MicroSD card (minimum 4GB)
• Micro USB power supply (2.5A). For Raspberry Pi 4, you need the USB-C power supply
• USB keyboard and mouse
• Full-size HDMI cable to connect Raspberry Pi to a monitor or TV duing the initial setup. For Raspberry Pi 4, you need the Micro HDMI cable
• Raspberry Pi case

Operating system

• Raspbian -  the offical Raspberry Pi operating system. I want to keep the OS footprint small and I don’t plan to connect to a monitor other than the initial setup, so I select Raspbian Lite image.

Basic procedure

1. Download Raspbian Lite image
2. Download balenaEtcher
3. Launch balenaEtcher and write the Raspbian image to the SD card
4. Insert the SD card, connect the network cable, the keyboard & mouse, monitor and power supply to the board
5. Power on and walk through the setup
6. At the console prompt, log in with the default credential: pi / raspberry
7. passwd
   • change the user pi’s password 
8. sudo raspi-config
   • configure time zone, locale and keyboard layout
   • enable SSH under Interfacing options
9. sudo nano /etc/dhcpcd.conf
   • configure a static IP address on eth0 
   • interface eth0
     static ip_address=x.x.x.x/24
     static routers=x.x.x.x
     static domain_name_servers=x.x.x.x
     
10. sudo nano /etc/sysctl.conf
    • disable IPv6, by adding a line “net.ipv6.conf.all.disable_ipv6 = 1"
    • sudo sysctl -p
11. sudo apt update && sudo apt upgrade -y
12. sudo reboot
13. after reboot, verify SSH login

Now the Raspberry Pi is fully functional. I’m going to use it to set up Pi-Hole to block Ad, DNS-Over-HTTPS (DoH) for privacy, CUPS to add AirPrint to an USB printer, and more. Stay tuned.

View and Modify Delete On Termination Setting on AWS EBS Volume

When launching an AWS EC2 instance, you can specify whether a EBS volume will be deleted when the instance is terminated. The setting is “Delete On Termination” and it’s on by default.

Once the instance is launched, viewing or modifying this “Delete On Termination” setting is not obvious. This post is to document how I do it.

View “Delete On Termination” Setting on AWS EBS Volume

There are two ways:

• Use EC2 management console
• Click on Instances and select the instance
• Under Description tab,scroll down to “Block devices"
• Click on the device name to show the “Delete on termination” setting on the pop-up window

• Use AWS CLI
  • Get the instance ID
    • $ aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --output text
  • Query the setting
    • aws ec2 describe-instances --filter "Name=instance-id,Values=i-0d0a326ab6691cf24" --query 'Reservations[*].Instances[*].BlockDeviceMappings[*].Ebs.DeleteOnTermination'

Modify “Delete On Termination” Setting on AWS EBS Volume

I cannot find a way to modify this setting on EC2 management console. It looks AWS CLI is the only way. Here is the CLI.

aws ec2 modify-instance-attribute --instance-id i-0d0a326ab6691cf24 --block-device-mappings file://./change.DeleteOnTermination.json

The json file format is below. Update the DeviceName to match the EBS volume that you wan to modify

[
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": false
}
}
]

After executing the CLI, refresh the management console and click on the volume name under the instance description to verify the setting has been modified.

Using growpart to extend a Linux non-LVM partition

This post is about using growpart to extend a non-LVM Linux partition. For extending a LVM volume, see my other posts - Extend a Linux LVM Volume on a VM part1, part2, and part3.

When increasing the size of a Linux parition, it normally requries the following procedures:

1. Increase the size of the physical or virtual hard drive
2. Extend the partition to added drive space
3. Resize the file system to extended partition

Among these steps, step #2 usually sounds risky. It requires to delete the existing partition and recreating it with the new size. Most people are not confortable to do that, and they end up with adding a new bigger drive, creating a new bigger partition, formatting the new partition, and copying the files from the old partition to the new bigger. This not only is time comsuming, but only requires more physical or vritual disk space.

Recently I learned about growpart that makes step #2 much simpler and error free. Growpart may not be installed by default, but it should be available on your distro. The following is the commands to increase a Linux partition (/dev/sdb1) on CentOS.

• sudo yum install cloud-utils-growpart
• increase the disk size (e.g. /dev/sdb) and reboot
• sudo growpart /dev/sdb 1 ### there is a space between sdb and 1. 1 is the partition number
• sudo resize2fs /dev/sdb1

See growpart man page for more info.

Installing vSphere Client fails with the error “VMInstallHcmon - Failed to install hcmon driver”

When installing the latest vSphere Client 6.0 on my Windows 10 computer, I got the following error “VMInstallHcmon – Failed to install hcmon driver”

Troubleshooting

• Try KB2006486. But I don’t see Non-Plug and Play Drivers and VMware hcmon on my Windows 10 computer
• Try renaming the C:\Windows\System32\drivers\hcmon.sys file. Still get the same error

Solution

• On my laptop, vSphere Client 5.5 and 6.0 (older build), and their respective Update Manager plug-in are installed
• Remove these older clients and plug-ins
• vSphere Client 6.0 installation completes successfully

vSphere Client 6.0 could not connect to vCenter server error “The client did not receive a complete response from the server”

Due to security concerns in the TLS v1.0, we start disabling it on the system. After disabling TLS v1.0 on a Windows machine, I got the following error when connecting to a vCenter Server 6.0 using the vSphere Client (legacy C# client).

Troubleshooting

• I am still able to login vCenter via the vSphere Web Client on the web browser
• Re-enabling TLS v1.0 on the machine, I am able to login vCenter via the vSphere Client again
• The vCenter server is a VCSA 6.0 Update 3e. TLS v1.1/v1.2 are enabled by default (KB2145796).

Solution

1. Download and install the latest vSphere Client 6.0. As of this writing, vSphere Client 6.0.0 build 6855219.
2. Navigate to the configuration file C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe.config
3. Edit the VpxClient.exe.config file by setting the parameters
   <add key = “EnableTLS12” value = “false” />
   as
   <add key = “EnableTLS12” value = “true” />

Additional notes

• An older build of the vSphere Client 6.0 may not have the “EnableTLS12” entry in the configuration file. Install the latest build from the link in step #1.
• Some Microsoft updates or .NET 4.6.2 may need to install on the machine. See KB2149000.

Set Visual Studio Code (VS Code) Default Language Mode When New File Opened

Visual Studio Code is my default PowerShell script editor. It is free and available in Windows, macOS, and Linux.

By default, it uses text as the script language when a new file is opened. To make it recognize the PowerShell cmdlet, I have to manually change the language mode (press F1, type Change Language Mode, then select PowerShell form the list of the language) or first save to the file to ps1. This becomes inconvenience overtime.

Luckily, a new setting “files.defaultLanguage” is added. I didn’t backtrack when this setting was added. It’s available in the current release 1.14.0 as of this writing.

To configure the default language setting,

• Open File, Preferences, Settings
• Add the following to set PowerShell as the default language mode
• “files.defaultLanguage": "powershell",
• Save the setting.json file

To set other supported languages as the default

• Delete “powershell” in the line
• Press Ctrl + Spacebar to select the option from the list

Use WinSCP to Transfer Files in vCSA 6.5

To use WinSCP to transfer files with vCSA, VMware KB2107727’s solution is temporarily changing the default shell from appliancesh to bash, then changing back after the transfer. This works in vCSA 6.0 and vCSA 6.5.

In vCSA 6.0, there is a tick to change the WinSCP’s Advanced, SFTP server setting to "shell /usr/lib64/ssh/sftp-server" (without the quotes) for file transfer without changing the default shell. See “Connect to vCSA using WinSCP”

VMware changed the OS from SLES to Photon in vCSA 6.5. The above setting doesn’t work anymore. There is no ssh directory under /usr/lib64/. However, a quick search and find the sftp-server is moved to /usr/libexec/. Using the setting “shell /usr/libexec/sftp-server” in vCSA 6.5 works fine.

Here is the detail instruction.

• Log in vCSA VAMI UI (https://vcsa-ip:5480)
• Under Access, enable SSH Login. (PS: enabling Bash Shell is not necessary)
• Open WinSCP, select File protocol: SFTP
• Enter the vCSA hostname, port number 22, root, and root’s password
• Click Advanced
• Under Environment, SFTP, Protocol options
• Set SFTP server to “shell /usr/libexec/sftp-server” (without the quotes)

PS: I tested the setting in WinSCP v5.9.6 build 7601 and vCSA v6.5.0.5600 build 4951144.

VCSA 6.5 “The appliance management service is not running” Fix

Scenario

In vSphere Web Client 6.5, under Home, Administration, Deployment/System Configuration, Nodes, the vCenter Server node shows an error message “The appliance management service is not running”. An error message “HTTP response with status code 503, 503 Service Unavailable (Failed to connect to endpoint: _serverNamespace = /vmonapi action =Allow _port = 8900" also appears in the web client.

Troubleshooting

• Login the VMware Appliance Management UI (https://psc:5480 or https://vc:5480). All the health status are good.
• SSH to VC appliance. Check service status (KB2109887)
• # service-control –list
• # service-control –status
• applmgmt (VMware Appliance Management Service) is running
• vmonapi (VMware Service Lifecycle Manager API) is not running

Solution

• Restart vmonapi service or restart all services
• # service-control --start vmonapi
• # service-control –start –all
• PS: if restarting all the services, it may take some time before all services turn back to Good (green) in the node’s Summary page. e.g. VMware Performance Charts service takes more than 30 minutes to change from Warning, Unknown, and then Good.

VCSA 6.5 Syslog vs vRLI’s vSphere Integration

I write this post after reading William Lam’s “What logs do I get when I enable syslog in VCSA 6.5?” and doing some of my experiment on my VCSA 6.5 and vRLI 4.5 setup.

Background

Recently I completed a fresh VCSA 6.5 (external PSC and VC) deployment with vRealize Operations Manager (vROPS) 6.6 and vRealize Log Insight 4.5 installation. In vROPS, I configured vSphere and vRLI solutions; in vRLI, I configured vSphere and vROPS integration. I thought I completed all the setup until reading William’s blog post.

Confusion

There are a lot of information on his blog post. I was a little lost at the beginning, and I was wondering: should I configure VCSA syslog to vRLI? Is the same as vRLI’s vSphere integration? If I read his blog carefully, I would find the answer there. I didn’t fully understand it until I did my own experiment. Here is the quote. I highlighted a few key points.

I personally think the vSphere Integration is a nice solution if you have both Windows vCenter Server and the VCSA and to be able to get data consistency between the two platforms from a logging standpoint. It is definitely useful if you need to quickly enable all ESXi hosts connected to the vCenter Server and have them remotely syslog to the vRLI instance. If you only have the VCSA, you would get more information by configuring the remote syslog capability in VCSA rather than using the vSphere integration feature of vRLI. This especially true if you need the vpxd.log which is generally required for troubleshooting and debugging vCenter Server issues when calling into VMware Support. The other added benefit to using the VCSA option is that structure log entries are processed directly on the VCSA rather than having to be remotely queried via the vSphere APIs, processed and then store in vRLI which would add additional load onto vRLI, especially if you need to configure additional vCenter Server instances.

Summary

I summarize based on my understanding of this topic here. Please refer his blog for the full details.

• VCSA 6.5 has a new remote syslog functionality comparing to VCSA 6.0. This function is not available in Windows vCenter Server 6.5
• VCSA 6.5’s remote syslog configuration is in the VAMI UI (https://[VCSA]:5480). This setting available in both PSC and VC for external deployment. See William’s post’s “Logs forwarded by VCSA Deployment Type” for the logs forwarded in different VCSA deployment type
• VCSA 6.0’s remote syslog configuration is in the vCenter via vSphere Web Client
• VCSA 6.5 has a new Enhanced Logging feature (see William’s blog for what the enhanced means; see my screen shots in this post for a better example)
• After completing vRLI’s vSphere integration, “enable streaming of events to syslog” is enabled (vSphere Web Client, vCenter, Configure, Advanced Settings, vpxd.event.syslog.enabled). This setting is mentioned in another person blog. I am not sure what the default VCSA setting is. Put it here for the reference only
• VCSA 6.5 remote syslog is not configured even completing vSphere integration in vRLI
• VCSA 6.5 remote syslog is “pushing” the logs to vRLI
• vRLI’s vSphere integration is “pulling” the logs from VCSA (via vSphere API). This supports both a Windows vCenter Server and VCSA.
• vRLI’s vSphere integration can also automatically configure the ESXi hosts connected to the vCenter Server and have them remotely syslog to vRLI. (vSphere Web Client, ESXi host, Configure, System/Advanced System Settings, Syslog.global.logHost)
• By default, vCenter Server log (vpxd.log) is not forwarded to a remote syslog server. It is recommended enabling it for troubleshooting purposes. (vSphere Web Client, vCenter, Configure, Advanced Settings, config.log.outputToSyslog; then restart vCenter Server service in System Configuration, Services, VMware vCenter Server)
• Other VCSA 6.5 logs can be forwarded to a remote syslog server. but it’s not supported by VMware. See the link at the end of William’s post for more details
• This is the most important and useful point I have learned. VCSA 6.5 remote syslog sends more information to vRLI comparing to vRLI’s integration. I think this is what the Enhanced Logging means. See my screen shots below. For example, I modified the Tools Upgrades option on a VM.
• Without VCSA remote syslog configured, vRLI has one entry in the log. It shows the name of the VM (highlighted in yellow)’s toolsUpgradePolicy is changed from “manual” to “upgradeAtPowerCycle”
• With VCSA remote syslog configured, vRLI has two entries in the log. In additional to the regular log, the second entry shows the name of the user made the change (highlighted in the red box).
• My recommendation is to configure both vRLI’s vSphere integration (for automate configuring the ESXi log host) and VCSA remote syslog (for the enhanced logging). This would duplicate some log entries in vRLI and consume more vRLI log storage. But it is well worthy!

vSphere 6.5 New Feature – VMware Orchestrated Restart

Let me back to the old ESXi 3 day – when I was just using the standalone ESXi hosts or vCenter without HA and DRS. In case of the power outage or air conditioning failure in the data center, all the ESXi hosts were powered down. Once the environment problem was resolved, I could manage the VM startup sequence by configuring the switched PDU to start the hosts accordingly, and configuring the VM startup order at the host level.

However, once I deployed vCenter Server with HA and DRS, I lost the control of the VM startup order. Because the VMs could be hosted at any host in the cluster. Someone said that I should not worry about the VM startup order in the cluster. Because the ESXi cluster would never go down if I had designed the infrastructure with enough redundancy. As we all know, we never have enough redundancy in a small ESXi deployment.

I have been curious why VMware do not “fix” this issue for so long. Until now, vSphere 6.5 introduces the VMware Orchestrated Restart feature. At the high level, the Orchestrated Restart, likes the VM affinity and anti-affinity rules, put the VMs in different VM groups and set the startup dependence among the VM groups. To learn more about this, please go to “What is VMware Orchestrated Restart?”.

I am so glad to know about this new vSphere 6.5 feature – one more reason to upgrading to vSphere 6.5.