Search This Blog

GPO Security Setting Persistence

For security compliance purpose, I want to enforce some Windows settings on the Windows server via the group policy. Some GPOs already exist in the domain. Instead of modifying the existing GPOs, I create a new GPO and link it to the OU where the servers locate. I think I can easily roll back / undo the new settings if they cause any issue, by moving the server out of the OU or unlinking the GPO to the OU.

It turns out my original thought is not 100% correct. Some security settings still persist even if the setting is no longer defined in the policy.

For example, on a new Windows Server 2012 R2 server, the local security policy setting of “Network Security: LAN Manager authentication level” is “Not Defined”. In my new GPO, I configure this setting to “Send NTLMv2 response only; Refuse LM & NTLM”. After linking this new GPO to the OU, I cannot RDP to the servers with the domain login (the error message is “The logon attempted failed”, even my username and password are correct), but the local login is okay.

I unlink the new GPO from the OU for troubleshooting and expect the security settings rolling back to their original configuration. However, I get the same error. And the local security policy on the server still configures “Send NTLMv2 response only; Refuse LM & NTLM”. I reset this setting on the local security policy back to the default “Not Defined”. That solves the problem.

Some posts give me the idea on the fix: post 1, post 2.

Now I learn the “tattooing” behavior on the security settings in the GPO.

Persistence in security settings

Security settings may still persist even if a setting is no longer defined in the policy that originally applied it.

Persistence in security settings occurs when:

  • The setting has not been previously defined for the computer.

  • The setting is for a registry object.

  • The setting is for a file system object.

All settings applied through local policy or a Group Policy Object are stored in a local database on your computer. Whenever a security setting is modified, the computer saves the security setting value to the local database, which retains a history of all the settings that have been applied to the computer. If a policy first defines a security setting and then no longer defines that setting, then the setting takes on the previous value in the database. If a previous value does not exist in the database, then the setting does not revert to anything and remains defined as is. This behavior is sometimes called "tattooing."

Registry and file settings will maintain the values applied through policy until that setting is set to other values

Reference: Administer Security Policy Settings, “Persistence in security settings”

How to Convert Eager Zeroed Thick Disk to Lazy Zeroed Think Disk

According to the VMware KB2145183, an Eager Zeroed Thick disk cannot be directly converted / migrated / cloned to Lazy Zeroed Think disk. The work around is:

  1. Convert the Eager Zeroed Think disk to Thin
  2. Convert the Thin disk to Lazy Zeroed

The vmkfstools can be used for the conversion (see my previous post). Here is the commands:

  1. vmkfstools -K <vmdkFile>
  2. vmkfstools -j <vmdkFile>

Error “Idm client exception: Error trying to join AD, error code [11]” when joining a VCSA to AD domain

On a newly created VCSA appliance, I got the following error when joining it to the Active Directory domain

vcsa.joining.ad.error

I used the domain’s netbios name\user-name as the user name.

Fix: use the User Principal Name (UPN), user-name@fqdn-domain-name, as the user name. After joining the domain, reboot the VMware Platform Service Controller (PSC).

Troubleshoot vMotion Error 195887167

Updated on 07/13/2016. See the update this post, I might find the ultimate solution, even I am still not sure what the cause of the issue.

Recently I had an issue to vMotion some VMs between the vSphere v 6.x cluster hosts. Long story short, here are the symptoms:

  • I consistently got the error “Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout” when vMotioning (host only, no storage vMotion) on some VMs, particularly the vCenter Server Appliance VM. I had two vCSA VMs. Both were have the same issue. But vCSA is not the only VM that I got the error.
  • I successfully vMotion some VMs between the hosts. The vMotion network configuration should be okay.
  • In other words, some VMs are okay; some are not. The size (CPU, RAM, storage) of the VMs does not seem the problem. The successfully vMotioned VMs can have more/less CPU, RAM, storage than the failed VMs.
  • The failed VM has more disks than other VMs. The vCSA VM created 11 disks by default.
  • When the VMs were powered off, vMotion successfully.
  • Restarted the hosts and restarted the VMs. No difference.
  • Verified no IP address conflicts.
  • Tried one VMkernel adapter for both management and vMotion or a dedicated VMkernel adapter for vMotion. No difference.
  • Tested vmkping successfully between the hosts.
  • In the vmkernel.log of the hosts, the error is “2016-05-18T22:47:34.959Z cpu15:39089)WARNING: Migrate: 270: 1463611379229538 D: Failed: Connection closed by remote host, possibly due to timeout (0xbad003f) @0x418000e149ee” on the destination host or “2016-05-19T18:29:23.930Z cpu1:130133)WARNING: Migrate: 270: 1463682486286991 S: Failed: Migration determined a failure by the VMX (0xbad0092) @0x41803a7f6993” on the source host.

Possible solutions

  • Remove the snapshot on the VM if it has one
    • After removing the snap shot on one of vCSA VMs, vMotion worked fine. But another vCSA had no snapshot, it still failed.
  • Try using the vSphere Client instead of the vSphere Web Client. This worked on some VMs, but not always.
  • Assign VM’s network adapter to different port group; and change back to its original port group
    • This seems the ultimate fix. After doing this, the vCSA VMs, which failed vMotion consistently, are vMotioned successfully.

Conclusion

07/13/2016 update

  • I run into the same error when migrating some VMs, including the vCenter 6.x appliance (vCSA), between hosts in the Vmware cluster. I am able to migrate other VMs. This leads me to believe the problem on the VM, instead of the VM infrastructure.
  • Within the VMs having this error, some of them, which I can power off, are migrated successfully. However, I can not power down the vCSA VM. Because I cannot perform the migration without the vCenter available.
  • I try assigning the NIC of the vCSA VM to different port group, and change back. However, I cannot do that, because this VM is the vCenter server and is configured with a distributed switch. If I change the vCSA to different port group (configured with a different VLAN), the vCenter server will be down (because its NIC is assigned to the wrong port group with the wrong VLAN); and I cannot change it back to the original port group with the right VLAN.
  • I try connecting directly to the ESXi host of the vCSA VM via the vSphere C# client, then assign the NIC of the vCSA VM to different port group. However, I cannot do that either. Because the host is only configured with the distributed switches, there is no other port group in the selection at this situation.
    • vmotion.error.01
  • I try create a new port group with the ephemeral port binding on the distributed switch. This new ephemeral port group is available in the vSphere C# client when connecting to the host directly. Then I assign the VM to the ephemeral port group and change back. However, the migration still fails with the same error.
  • Since I could fix this issue last time by changing the port group and changing back, I guess that somehow reset the NIC on the VM or the virtual switch port to which the VM is connected. That gives me an idea to manually assign the VM to another virtual switch port.
    • vmotion.error.02

Solutions

  • In the screenshot above, the NIC of the vCSA is assigned to port 214 on the vSwitch.
  • Log back in the vCenter via the vSphere C# client or Web client (I cannot see the ports on the distributed switch, nor change the port assigned to the VM’s NIC when connecting to the ESXi host via the vSphere C# client)
  • I find an used port on the same port group (e.g. port 423 in my case).
    • vmotion.error.03
  • Edit the vCSA VM setting and assign its NIC to the unused port.
  • Then I can successfully migrate the vCSA VM to another host.

Conclusion

  • Changing the NIC to an used port will be my first attempt when this issue happens again (I bet it will happen).
  • I still don’t know the cause of the this issue.

10/15/2019 update

  • I got the exact error again when Storage vMotion a VM across two vCenter (from vCSA 6.0 to vCSA 6.5). Deleted the VM snapshot, reran the vMotion, and completed successfully.

Change Windows Server 2008 or 2012 Network Profile

Sometimes a Windows server is assigned to the incorrect network profile. It can cause applying the wrong Windows Firewall rules. Here is how to change its network profile.

For standalone server

  • Can change the profile to public or private; but cannot set to domain
  • For Windows Server 2012
    • Open PowerShell as administrator
    • Get-NetconnectionProfile | Set-NetconnectionProfile -NetworkCategory [Private | Public]
  • For Windows Server 2008 or 2012
    • gpedit.msc, Computer Configuration, Windows Settings, Security Settings, Network List Manager Profiles
    • Select the network name, Properties, Network Location
    • Under Location Type, select Private or Public

For domain joined server

Extend Microsoft Cluster Shared Disk in VMware

A VM shared disk on Microsoft Cluster Service (MSCS) is running out of disk space. The VMs are on a single host (aka cluster in a box - CIB). I can think of two ways to expand the disk storage.

  • create a new big shared disk for the cluster, migrate the data, then change the new disk to the same drive letter as the original disk
  • extend the size of the existing shared disk

Obviously the latter seems simpler, but it requires special attention. The shared disk format in MSCS VMs must be in eager zeroed thick format. However, when extending an eagerzeroedthick VMDK, the extended chuck is in lazy zeroed thick format by default (reference “Extending an EagerZeroedThick Disk”. In my test, vSphere 6 has the same behavior)

Here is how I extend the MSCS shared disk

  • Power off both servers in the cluster
  • Increase the VMDK disk size. There are two ways:
    • GUI: edit the VM settings, increase the shared disk size
    • CLI: use vmkfstools -X <newsize> -d eagerzeroedthick <vmdkfile>
  • Using the GUI, the extended chuck will be in lazy zero thick format. The VM will fail to power on with the error “VMware ESX cannot open the virtual disk for clustering…”

cluster.vm.power.on.error

  • There are two ways to convert the extended chuck to eagerzeroedthick format
    • Migrate the VM to another storage, and specify the eager zero thick format for the disk
    • Use vmkfstools -k <vmdkfile>
      vmkfstools.convert.eagerzeroedthick
  • Once the entire shared disk is the eager zeroed thick format, the VM will be able to power on.
  • Extend the Windows partition as KB304736

vmkfstools Examples

When searching an issue on expanding a shared disk on Microsoft clustering VMs (CIB), I have learned more about the vmkfstools command.

The vmkfstools --help displays many options, but lack of explanation. So I document them here. (reference: vSphere Storage, Using vmkfstools)

# vmkfstools --help

OPTIONS FOR FILE SYSTEMS:

vmkfstools -C --createfs [vmfs3|vmfs5]
               -b --blocksize #[mMkK]
               -S --setfsname fsName
           -Z --spanfs span-partition
           -G --growfs grown-partition
   deviceName

           -P --queryfs -h --humanreadable
           -T --upgradevmfs
   vmfsPath
           -y --reclaimBlocks vmfsPath [--reclaimBlocksUnit #blocks]

OPTIONS FOR VIRTUAL DISKS:

vmkfstools -c --createvirtualdisk #[gGmMkK]
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               ]
               -a --adaptertype [buslogic|lsilogic|ide
                                |lsisas|pvscsi]
               -W --objecttype [file|vsan]
               --policyFile <fileName>
           -w --writezeros
           -j --inflatedisk
           -k --eagerzero
           -K --punchzero
           -U --deletevirtualdisk
           -E --renamevirtualdisk srcDisk
           -i --clonevirtualdisk srcDisk
               -d --diskformat [zeroedthick
                               |thin
                               |eagerzeroedthick
                               |rdm:<device>|rdmp:<device>
                               |2gbsparse]
               -W --object [file|vsan]
               --policyFile <fileName>
               -N --avoidnativeclone
           -X --extendvirtualdisk #[gGmMkK]
               [-d --diskformat eagerzeroedthick]
           -M --migratevirtualdisk
           -r --createrdm /vmfs/devices/disks/...
           -q --queryrdm
           -z --createrdmpassthru /vmfs/devices/disks/...
           -v --verbose #
           -g --geometry
           -x --fix [check|repair]
           -e --chainConsistent
           -Q --objecttype name/value pair
           --uniqueblocks childDisk
   vmfsPath

OPTIONS FOR DEVICES:

           -L --lock [reserve|release|lunreset|targetreset|busreset|readkeys|readresv
                     ] /vmfs/devices/disks/...
           -B --breaklock /vmfs/devices/disks/...

vmkfstools -H --help

vmkfstools Command Syntax

vmkfstools options target

Options: separate into three types - File System Options, Virtual Disk Options, and Storage Device Options.
Target: partition, device, or path

File System Options

  • Listing Attributes of a VMFS Volume
    The listed attributes include the file system label, if any, the number of extents comprising the specified VMFS volume, the UUID, and a listing of the device names where each extent resides.
    vmkfstools -P -h <vmfsVolumePath>
    vmkfstools -P -h /vmfs/volumes/netapp_sata_nfs1/
  • Creating a VMFS Datastore
    vmkfstools -C vmfs5 -b <blocksize> -S <datastoreName> <partitionName>
    vmkfstools -C vmfs5 -b 1m -S my_vmfs /vmfs/devices/disks/naa.
    ID:1
  • Extending an Existing VMFS Volume
    vmkfstools -Z <span_partition> <head_partition>
    vmkfstools -Z /vmfs/devices/disks/naa.disk_ID_2:1 /vmfs/devices/disks/naa.disk_ID_1:1
    Caution: When you run this option, you lose all data that previously existed on the SCSI device you specified in span_partition.
  • Growing an Existing Extent
    vmkfstools –G device device
    vmkfstools --growfs /vmfs/devices/disks/disk_ID:1 /vmfs/devices/disks/disk_ID:1

Virtual Disk Options

  • Creating a Virtual Disk
    vmkfstools -c <size> -d <diskformat> <vmdkFile>
    vmkfstools -c 2048m testdisk1.vmdk
  • Initializing a Virtual Disk
    vmkfstools -w <vmdkFile>
    This option cleans the virtual disk by writing zeros over all its data. Depending on the size of your virtual disk and the I/O bandwidth to the device hosting the virtual disk, completing this command might take a long time.
    Caution: When you use this command, you lose any existing data on the virtual disk.
  • Inflating a Thin Virtual Disk
    vmkfstools -j <vmdkFile>
    This option converts a thin virtual disk to eagerzeroedthick, preserving all existing data. The option allocates and zeroes out any blocks that are not already allocated.
  • Removing Zeroed Blocks (Converting a virtual disk to a thin disk)
    vmkfstools -K <vmdkFile>
    Use the vmkfstools command to convert any thin, zeroedthick, or eagerzeroedthick virtual disk to a thin disk with zeroed blocks removed.
    This option deallocates all zeroed out blocks and leaves only those blocks that were allocated previously and contain valid data. The resulting virtual disk is in thin format.
  • Converting a Zeroedthick Virtual Disk to an Eagerzeroedthick Disk
    vmkfstools -k <vmdkFile>
    Use the vmkfstools command to convert any zeroedthick virtual disk to an eagerzeroedthick disk. While performing the conversion, this option preserves any data on the virtual disk.
  • Deleting a Virtual Disk
    vmkfstools -U <vmdkFile>
    This option deletes files associated with the virtual disk listed at the specified path on the VMFS volume.
  • Renaming a Virtual Disk
    vmkfstools -E <oldName> <newName>
  • Cloning or Converting a Virtual Disk or Raw Disk
    cloning:
    vmkfstools -i <sourceVmdkFile> <targetVmdkFile>
    vmkfstools -i /vmfs/volumes/templates/gold-master.vmdk /vmfs/volumes/myVMFS/myOS.vmdk
    converting: vmkfstools -i <sourceVmdkFile> -d <diskfomrat> <targetVmdkFile>
  • Extending a Virtual Disk
    vmkfstools -X <newSize> [-d eagerzeroedthick] <vmdkFile>
    use -d eagerzeroedthick to ensure the extended disk in eagerzeroedthick format.
    Caution: do not extend the base disk of a virtual machine that has snapshots associated with it. If you do, you can no longer commit the snapshot or revert the base disk to its original size.
  • Displaying Virtual Disk Geometry
    vmkfstools -g <vmdkFile>
    The output is in the form: Geometry information C/H/S, where C represents the number of cylinders, H represents the number of heads, and S represents the number of sectors.
  • Checking and Repairing Virtual Disks
    vmkfstools -x <vmdkFile>
    Use this option to check or repair a virtual disk in case of an unclean shutdown

Storage Device Options

  • Managing SCSI Reservation of LUNs
    Caution: Using the -L option can interrupt the operations of other servers on a SAN. Use the -L option only when troubleshooting clustering setups.
    • vmkfstools -L reserve <deviceName>
      Reserves the specified LUN. After the reservation, only the server that reserved that LUN can access it. If other servers attempt to access that LUN, a reservation error results
    • vmkfstools -L release <deviceName>
      Releases the reservation on the specified LUN. Other servers can access the LUN again
    • vmkfstools -L lunreset <deviceName>
      Resets the specified LUN by clearing any reservation on the LUN and making the LUN available to all servers again. The reset does not affect any of the other LUNs on the device. If another LUN on the device is reserved, it remains reserved
    • vmkfstools -L targetreset <deviceName>
      Resets the entire target. The reset clears any reservations on all the LUNs associated with that target and makes the LUNs available to all servers again.
    • vmkfstools -L busrest <deviceName>
      Resets all accessible targets on the bus. The reset clears any reservation on all the LUNs accessible through the bus and makes them available to all servers agai
    • When entering the device parameter, use the following format:
      /vmfs/devices/disks/vml.vml_ID:P

Hidden Options (reference: “Some useful vmkfstools ‘hidden’ options”)

  • VMDK Block Mappings
    vmkfstools -t0 <vmdkFile>
    Display the chuck file format in a VMDK file.
    • VMFS -- = eager zeroed thick
    • VMFS Z- = lazy zeroed thick
    • NOMP -- = thin

Use WinSCP to Transfer Files in vCSA 6.7

This is a quick update on my previous post “ Use WinSCP to Transfer Files in vCSA 6.5 ”. When I try the same SFTP server setting in vCSA 6.7...