Search This Blog

Use \\?\UNC\ on A UNC Name to Work Around Windows 260 Characters File Name Limit

Prefixing the file name with “\\?\” can work around the Windows 260 characters (up to 32,000 characters) file name limit in some command line utilities. For example, for local file, dir \\?\c:\longfilename.txt.

However, the syntax is different on a UNC name, it needs to be \\?\UNC\servername\share\longfilename.txt. For example, dir \\?\UNC\fileserver\dept\it\longfilename.txt.

Not all the command line utilities support this syntax.

  • Icacls - support
  • PowerShell (get-acl) - not support, because of .NET

For more info see

Long Paths in .NET, Part 1 of 3

Remove OneDrive from File Explorer Side Panel in Windows 10

On my work PC, I use OneDrive for Business instead of OneDrive. By default Windows 10 includes an OneDrive shortcut in File Explorer side panel.

OneDrive.01OneDrive.02

Change the following registry key to remove it.

  • {HKEY_CLASSES_ROOT\CLSID\{018D5C66-4533-4307-9B53-224DE2ED1FE6}
  • In the subkey System.IsPinnedToNameSpaceTree change the DWORD from 1 to 0
  • No reboot required.

Source: Microsoft TechNet Forums

Use Sigcheck and VirusTotal to Scan Files for Malware

Sigcheck is a tool to check files digital signature, and recently is updated to auidt the root CA stores. And it can also be used to scan files for malware via VirusTotal.

Unless the -vs option is used, sigcheck only compares the hash of a file in VirusTotal’s database to determine whether the file contains malware.

  • Download the latest version of Sigcheck (as of 01/26/2016, v.2.42)
  • Unzip the zip file to a folder
  • Open Command Prompt as Administrator, and change to the folder
  • Run sigcheck.exe -vt, and enter y to agree the VirusTotal policy
  • To scan a file: sigcheck.exe -v <filename>
  • To upload a file that is not previously scanned to VirtualTotal; the file size is up to 128MB: sigcheck.exe -vs <filename>
  • To scan a folder and subfolder: sigcheck.exe -v -s <foldername>
  • To scan and create a CSV report: sigcheck.exe -v -s -c <foldername> > <report.csv>
    • Check the VT detection column for the number of antivirus engine deteted a malware and used
    • Open the URL in VT link in a browser to get the detail of the analysis - which antivirus engine and type of malware

Fix A VSAN Host Shows 0 of 0 Disks In Use

We have three hosts running on VSAN 6.1. Today the Disk Management in vSphere Client shows one of the hosts 0 of 0 Disk in Use.

VSAN.Disk.Missing.01

And in VSAN General, it shows the warning of Mixed On-disk Format Version, and there is an upgrade button next to it. (Do Not Click It - I didn’t click it, and am not sure what the impact would be). Because our VSAN environment is built from scratch with VSAN 6.1, it is not upgrade from VSAN 5.5. It does not make sense the disk format requires an upgrade.

VSAN.Disk.Missing.02

Troubleshoot

  • Run VSAN Health check, everthing is green.

VSAN.Disk.Missing.03

  • The affected host shows all the disks under its Manage, Storage, Storage Devices.

VSAN.Disk.Missing.04

Solution

  • Click the first icon under Storage Devices to refresh the host’s storage information.

VSAN.Disk.Missing.05

Now the Disk Management and On Disk Fromat are back to normal.

VSAN.Disk.Missing.06

VSAN.Disk.Missing.07

Configure ESXi Network Dump Collector

When booting the ESXi from a SD, you probably need to reconfigure the ESXi dump collector location to a persistent datastore or a network dump collector.

The reason is the ESXi installer puts the scratch partition in “/tmp/scratch” on the local ramdisk. see the quote below from Booting ESXi off USB/SD.

3.  Where does the scratch partition get placed when booting from USB?

Because USB/SD devices are sensitive to high amounts of I/O the installer will not place the scratch partition on a USB/SD device.  Instead, the installer first scans for a local 4GB vfat partition, if it doesn’t find one it will then scan for a local VMFS volume on which to create a scratch directory.  If no local vfat partition or VMFS volume is found, as a last resort the installer will put the scratch partition in “/tmp/scratch” (i.e. put scratch on the local ramdisk).  If this happens it’s a good idea to manually reconfigure the scratch partition after the install.

The persistent store can be any available datastore (NFS, FC, iSCSI, local), except the VSAN datastore. If the ESXi host is a VSAN host, it’s likely you need to use the network dump collector instead of the persistent datastore.

There are two parts to set up the network dump collector:

  1. On the VCSA: Enable VMware vSphere ESXi Dump Collector service via vSphere Web Client
    • Administration, System Configurations, Services, VMware vSphere ESXi Dump Collector
    • Actions, Edit Startup Type, Automatic
    • Actions, Start
    • Note: the coredump file location is /var/core/netdumps
  2. On each ESXi host:
    • SSH to the ESXi host
    • esxcli system coredump network get
    • esxcli system coredump network set --interface-name <vmk0> --server-ipv4 <VCSA-IP-Address> --server-port 6500
    • esxcli system coredump network set --enable true
    • esxcli system coredump network check
      • or check the VCSA log file /var/log/vmware/netdumper/netdumper.log
    • /sbin/auto-backup.sh
      • to save the configuration file to persist after a reboot

See more info from “Booting ESXi off USB/SD”, KB2002955, Configure and Test of ESXi Dump Collector.

Use Telnet to Send HTTP Request

  • Launch Command Prompt
  • telnet <web-server> <port>
  • HEAD / HTTP/1.0
  • press Enter twice

Or

  • Launch Command Prompt
  • telnet <web-server> <port>
  • GET / HTTP/1.1
  • host: <web-server-hostname>
  • press Enter twice

Note: the HTTP commands in bold are case-sensitive, must be entered in the uppercase.

Removing Snapshots Can Cause VM Unresponsive

The first thing to remember is not to keep a VM snapshot for a long time – e.g. a few days for a busy VM. Because it

  • Can impact the VM performance and
  • Can cause the VM unresponsive when removing the snapshot (see KB1002836)

The second thing to remember is that removing or consolidating VM snapshots (particular a VM with a large snapshot file) when the VM is not busy.

Roll Back to A Previous Version of ESXi

Here are the steps to roll back to a perviouse version of ESXi: (source: KB1033604)

  1. Reboot the ESXi host
  2. When the hypervisor progress bar starts loading, press Shift + R.
  3. On the pop-up warning message “Current hypervisor will permanently be replaced with build: X.X.X-XXXXXX. Are you sure? [Y/n]”
  4. Press Shift + Y to roll back the build
  5. Press Enter to boot

Extend a Logical Drive in Windows Server 2008 R2

I recently successfully P2V a Windows Server 2008 R2 physical server to a vSphere VM. The server has two drives (Drive C and D), and both of drives were originally on one RAID 5 LUN. I didn’t know how the drives were originally created on the physical server, e.g.  whether Drive D is a primary partition or a logical drive on an extended partition.

When the Drive D starts low on the free disk space, I thought it’s very simple to extend a drive since the server is running Windows Server 2008.

  • I increase the size of hard disk 1 on the VM (both Drive C and D are on the hard drive) while the server is on
  • In Wndows Disk Management, rescan disks and then the extra unallocated disk space is showed
  • At this point, I noticed Drive D is a logical drive on an extended parition. I normally create the drive on the primary partition.
  • When right-click on the logical drive, the “Extend Volume” option is grayed out.
  • I had extended a non-OS drive in Windows Server 2008 via GUI before; and for Windows Server 2003, I had to use the diskpart utility.
  • Why I cannot extend this drive? Is it because it’s a logical drive on an extended partition? I’m not sure.
  • I know there are some third-party utilities can resize the partition. But I don’t have them handy.
  • Finally, I want to try the diskpart utility.
    • First, I tried extending the extended partition. It didn’t work. (see the yellow highlight)
    • Next, I tried extending the logical partition directly. It worked. (see the green highlight)

    C:\Windows\system32>diskpart

    Microsoft DiskPart version 6.1.7601
    Copyright (C) 1999-2008 Microsoft Corporation.
    On computer: AVLDB

    DISKPART> list disk

      Disk ###  Status         Size     Free     Dyn  Gpt
      --------  -------------  -------  -------  ---  ---
      Disk 0    Online          360 GB  1024 KB

    DISKPART> select disk 0

    Disk 0 is now the selected disk.
     
    DISKPART> list partition

      Partition ###  Type              Size     Offset
      -------------  ----------------  -------  -------
      Partition 1    OEM                 40 MB  1024 KB
      Partition 2    Primary           3072 MB    41 MB
      Partition 3    Primary             60 GB  3113 MB
      Partition 0    Extended           200 GB    63 GB
      Partition 4    Logical            200 GB    63 GB

    DISKPART> select partition 0

    Partition 0 is now the selected partition.

    DISKPART> extend

    There is no volume selected.
    Please select a volume and try again.

    DISKPART> list partition

      Partition ###  Type              Size     Offset
      -------------  ----------------  -------  -------
      Partition 1    OEM                 40 MB  1024 KB
      Partition 2    Primary           3072 MB    41 MB
      Partition 3    Primary             60 GB  3113 MB
    * Partition 0    Extended           200 GB    63 GB
      Partition 4    Logical            200 GB    63 GB

    DISKPART> select partition 4

    Partition 4 is now the selected partition.

    DISKPART> extend

    DiskPart successfully extended the volume.

NetScaler Packet Forwarding Flowchart

Source: Configuring Modes of Packet Forwarding

NetScaler Topologies Comparison

Source: Understanding Common Network Topolgies

Topology Two-Arm (inline)
Multiple Subnets Transparent
Client/Server IP on the different subnets on the same subnet
VIP public subnet no VIP
SNIP private subnet n/a
NSIP private subnet public subnet
MIP n/a public subnet
Server IP private subnet public subnet, configure the default gateway as the MIP
Layer 2 Mode n/a must enable
Use SNIP Option must enable n/a
Others the most commonly used topology if the clients need to access the servers directly
NetScaler is placed between the client and the server
Diagram
clip_image002[6]
clip_image004[6]
Task Overiew Task overview: To deploy a NetScaler appliance in two-arm mode with multiple subnets

1. Configure the NSIP and default gateway, as described in "Configuring the NetScaler IP Address (NSIP)."
2. Configure the SNIP, as described in "Configuring Subnet IP Addresses."
3. Enable the USNIP option, as described in "To enable or disable USNIP mode."
4. Configure the virtual server and the services, as described in "Creating a Virtual Server" and "Configuring Services."
5. Connect one of the network interfaces to a private subnet and the other interface to a public subnet.
Task overview: To deploy a NetScaler in two-arm, transparent mode

1. Configure the NSIP, MIP, and default gateway, as described in "Configuring a NetScaler by Using the Command Line Interface."
2. Enable L2 mode, as described in "Enabling and Disabling Layer 2 Mode."
3. Configure the default gateway of the managed servers as the MIP.
4. Connect the network interfaces to the appropriate ports on the switch.
Topology One-Arm
Single Subnet Multiple Subnets
Client/Server IP on the same subnet on the different subnets
VIP on the NetScaler on the NetScaler
SNIP n/a private subnet
NSIP public subnet private subnet
MIP public subnet n/a
Server IP public subnet private subnet
Layer 2 Mode n/a n/a
Use SNIP Option n/a must enable
Others connect one of the NICs to switch connect one of the NICs to switch
Diagram
clip_image002[8]
clip_image004[8]
Task Overiew Task overview: To deploy a NetScaler in one-arm mode with a single subnet

1. Configure the NSIP, MIP, and the default gateway, as described in "Configuring the NetScaler IP Address (NSIP)".
2. Configure the virtual server and the services, as described in "Creating a Virtual Server" and "Configuring Services".
3. Connect one of the network interfaces to the switch.
Task overview: To deploy a NetScaler appliance in one-arm mode with multiple subnets

1. Configure the NSIP and the default gateway, as described in "Configuring the NetScaler IP Address (NSIP)".
2. Configure the SNIP and enable the USNIP option, as described in "Configuring Subnet IP Addresses".
3. Configure the virtual server and the services, as described in "Creating a Virtual Server" and "Configuring Services".
4. Connect one of the network interfaces to the switch.

NetScaler Topologies Comparison

Source: Understanding Common Network Topolgies

Topology Two-Arm (inline) One-Arm
Multiple Subnets Transparent Single Subnet Multiple Subnets
Client/Server IP on the different subnets on the same subnet on the same subnet on the different subnets
VIP public subnet no VIP on the NetScaler on the NetScaler
SNIP private subnet n/a n/a private subnet
NSIP private subnet public subnet public subnet private subnet
MIP n/a public subnet public subnet n/a
Server IP private subnet public subnet, configure the default gateway as the MIP public subnet private subnet
Layer 2 Mode n/a must enable n/a n/a
Use SNIP Option must enable n/a n/a must enable
Others the most commonly used topology if the clients need to access the servers directly
NetScaler is placed between the client and the server
connect one of the NICs to switch connect one of the NICs to switch
Diagram
clip_image002
clip_image004
clip_image006
clip_image008

Extend a Linux LVM Volume on a VM - Part 3

This is the part 3 of extending a Linux LVM volume. see part 1 and part 2.

This part 3 is similar to part 2 when the partition as a PV. Instead of creating a new partition on the free disk space (like in part 2), delete the last partition on the disk and recreate it. This is useful when all the primary partitions (1 - 4) are already in use.

Extend a LVM when the partition as a PV

  1. Increase the VM’s hard disk size in vSphere Client

    • If there is a VM snapshot on the disk, its size cannot be changed. Remove all the snapshots first
    • After increasing the disk size, take a snapshot as the backup
  2. Rescan the SCSI bus to verify the OS see the new space on the disk
    • ls /sys/class/scsi_host/
    • echo “- - -“ > /sys/class/scsci_host/<host_name>/scan
    • tail -f /var/log/message
    • or
    • ls /sys/class/scsi_disk/
    • each ‘1’ > /sys/class/scsi_disk/<0\:0\:0\:0>/device/rescan
    • tail -f /var/log/message
    • fdisk -l
  3. Prepare the disk partition
    • fdisk -l
    • fdisk </dev/sdb>
    • p - print the partition table, note the last partition number in use
    • d - delete parition
    • p - primary partition
    • <X> - partition number, enter the last partition number from the previous p - print the partition table command 
    • n - add a new partition
    • p - primary partition
    • <X> - partition number, enter the partition number was deleted in the previous step 
    • default - the begining of the cylinder in the original partition
    • default - the last of the free cylinder
    • t - change a partition’s system id
    • <X> - partition number, enter the partition number was recreated in the previous step
    • 8e - Linux LVM
    • w - write table to disk and exit
    • fdisk -l to verify the new partition size
  4. Update partition table changes to kernel
    • reboot
    • or partprobe </dev/sdb>
    • Update (04/18/2016): In RHEL 6, partprobe will only trigger the OS to update the partitions on a disk that none of its partitions are in use (e.g. mounted). If any partition on a disk is in use, partprobe will not trigger the OS to update partition in the system because it it considered unsafe in some situations. So a reboot is required. see “How to use a new partition in RHEL6 without reboot?”
  5. Resize the PV
    • pvresize </dev/sdb3>
  6. Verifty the VG automatically sees the new space 
    • vgs
  7. Extend the LV
    • lvextend -l +100%FREE</dev/volume_group_name>/<logical_volume_name>
    • or lvextend -L+<size> /dev/<volume_group_name>/<logical_volume_name>
    • lvs
  8. Resize the file system
    • resize2fs /dev/<volume_group_name>/<logical_volume_name>
    • df -h
  9. Remove the VM snapshot once confirming the data intact

Extend a Linux LVM Volume on a VM - Part 2

This is the part 2 of extending a Linux LVM volume. See part 1 when the entire disk as a PV.

Extend a LVM when the partition as a PV

  1. Increase the VM’s hard disk size in vSphere Client

    • If there is a VM snapshot on the disk, its size cannot be changed. Remove all the snapshots first
    • After increasing the disk size, take a snapshot as the backup
  2. Rescan the SCSI bus to verify the OS see the new space on the disk
    • ls /sys/class/scsi_host/
    • echo “- - -“ > /sys/class/scsci_host/<host_name>/scan
    • tail -f /var/log/message
    • or
    • ls /sys/class/scsi_disk/
    • each ‘1’ > /sys/class/scsi_disk/<0\:0\:0\:0>/device/rescan
    • tail -f /var/log/message
    • fdisk -l
  3. Prepare the disk partition
    • fdisk -l
    • fdisk </dev/sdb>
    • p - print the partition table, note the next available partition number
    • n - add a new parition
    • p - primary partition
    • <X> - partition number, enter the next available partition number from the previous p - print the partition table command 
    • default - the begining of the free cylinder
    • default - the last of the free cylinder
    • t - change a partition’s system id
    • <X> - partition number, enter the next available partition number from the previous p - print the partition table command
    • 8e - Linux LVM
    • w - write table to disk and exit
    • fdisk -l to verify the new partition
  4. Update partition table changes to kernel
    • reboot
    • or partprobe </dev/sdb>
    • Update (04/18/2016): In RHEL 6, partprobe will only trigger the OS to update the partitions on a disk that none of its partitions are in use (e.g. mounted). If any partition on a disk is in use, partprobe will not trigger the OS to update partition in the system because it it considered unsafe in some situations. So a reboot is required. see “How to use a new partition in RHEL6 without reboot?”
  5. Initializ the disk partition
    • pvcreate </dev/sdb3>
  6. Extend the VG
    • use vgdispaly to determine the volume group name
    • vgextend <volume_group_name> </dev/sdb3>
    • vgs
  7. Extend the LV
    • lvextend -l +100%FREE</dev/volume_group_name>/<logical_volume_name>
    • or lvextend -L+<size> /dev/<volume_group_name>/<logical_volume_name>
    • lvs
  8. Resize the file system
    • resize2fs /dev/<volume_group_name>/<logical_volume_name>
    • df -h
  9. Remove the VM snapshot once confirming the data intact

Extend a Linux LVM Volume on a VM - Part 1

As I mentioned in the recent Linux LVM post, there are two ways to prepare the physical volume (PV)

  • the entire disk as a PV (not recommended)
  • or creating a partition on the disk and the partition as a PV.

The step to extend a LVM volume are different on these two configuration.

Extend a LVM when the entire disk as a PV

  1. Increase the VM’s hard disk size in vSphere Client
    • If there is a VM snapshot on the disk, its size cannot be changed. Remove all the snapshots first
    • After increasing the disk size, take a snapshot as the backup
  2. Rescan the SCSI bus to verify the OS see the new space on the disk
    • ls /sys/class/scsi_host/
    • echo “- - -“ > /sys/class/scsci_host/<host_name>/scan
    • tail -f /var/log/message
    • or
    • ls /sys/class/scsi_disk/
    • each ‘1’ > /sys/class/scsi_disk/<0\:0\:0\:0>/device/rescan
    • tail -f /var/log/message
    • fdisk -l
  3. Resize the PV
    • pvs
    • pvresize </dev/sdb>
    • image
  4. Verify the VG automatically sees the new space
    • vgs
    • image
    • compare the vg_app VFree size in this screen with the one in step #3
  5. Extend the LV
    • lvextend -l +100%FREE </dev/volume_group_name>/<logical_volume_name>
    • or lvextend -L+<size> /dev/<volume_group_name>/<logical_volume_name>
    • lvs
    • image
  6. Resize the file system
    • resize2fs /dev/<volume_group_name>/<logical_volume_name>
    • df -h
    • image
  7. Remove the VM snapshot once confirming the data intact

Linux Logical Volume Management (LVM) and Setup

LVM Layout

LVM Logical Volume Components

(source: RedHat Logical Volume Manager Administration)

LVM Components (from bottom to top)

  • Hard Disks
  • Partitions
    • LVM will work fine with the entire disk (without creating a partition) as a PV. But this is not recommended.
    • Other OS or disk utility (e.g. fdisk) will not recognize the LVM metadata and display the disk as free, so the disk is likely being overwritten by mistake.
    • The best pratice is to create a partition on the hard disk, then initialize the partition as a PV.
    • It is generally recommended that creating a single partition that covers the whole disk. (see RedHat Logical Volumen Manager Administration)
    • Using an entire disk a PV or using a partition as a PV will have a different procedure when growing the hard disk size in the VM (see “Expanding LVM Storage”)
  • Physical Volumnes
  • Volume Group
  • Logical Volumes
  • File Systems

LVM Setup

  1. Add a new hard disk
  2. Rescan the SCSI bus
    • ls /sys/class/scsi_host/
    • echo “- - -“ > /sys/class/scsci_host/<host_name>/scan
    • tail -f /var/log/message 
    • or
    • ls /sys/class/scsi_disk/
    • each ‘1’ > /sys/class/scsi_disk/<0\:0\:0\:0>/device/rescan
    • tail -f /var/log/message
  3. Prepare the disk partition
    • fdisk -l
    • fdisk </dev/sdb>
    • n - add a new parition
    • p - primary partition
    • 1 - partition number
    • default - first cylinder
    • default - last cylinder
    • t - change a partition’s system id
    • 1 - partition number
    • 8e - Linux LVM
    • w - write table to disk and exit
    • fdisk -l to verify the new partition
  4. Update partition table changes to kernel
    • reboot
    • or partprobe </dev/sdb>
    • Update (04/18/2016): In RHEL 6, partprobe will only trigger the OS to update the partitions on a disk that none of its partitions are in use (e.g. mounted). If any partition on a disk is in use, partprobe will not trigger the OS to update partition in the system because it it considered unsafe in some situations. So a reboot is required. see “How to use a new partition in RHEL6 without reboot?”
  5. Initialize disks or disk partitions
    • pvcreate </dev/sdb> - skip step #3, use the entire disk as a PV, not recommended
    • pvcreate </dev/sdb1> - use the partition created in step #3 as a PV, best practice
    • pvdisplay
    • pvs
  6. Create a volume group
    • vgcreate <volume_group_name> </dev/sdb1>
    • vgdisplay
    • vgs
  7. Create a logical volume
    • lvcreate --name <logical_volume_name> --size <size> <volume_group_name>
    • or lvcreate -n <logical_volume_name> -L <size> <volume_group_name>
    • lvdisplay
    • lvs
  8. Create the file system on the logical volume
    • mkfs.ext4 /dev/<volume_group_name>/<logical_volume_name>
  9. Mount the new volume
    • mkdir </mount_point>
    • mount /dev/<volume_group_name>/<logical_volume_name> </mount_point>
  10. Add the new mount point in /etc/fstab
    • vi /etc/fstab
    • /dev/<volume_group_name>/<logical_volume_name> </mount_point> ext4 defaults 0 0

Robocopy Command Examples

In my previous post, I summarize some key information about Robocopy. In this post, I will share some Robocopy command examples that I have used in the file migration projects. Even these examples have been successfully used on the production environment, please verify yourself before use. That is why I add the /L option on all the examples. Save the following to a batch file and pick the command by remove the rem at the beginning of the line.

@echo off

set LOG_FILE="C:\Temp\ROBOCOPY_LOG.TXT"
set SOURCE="Source Path"
set DEST="Destination Path"

rem Remove rem on the command line you want to run
rem Only one Robocopy command line is used at a time
rem Remove /L to apply the change after confirm the outcome in the log file

rem *************************************************
rem /PURGE option
rem Mirror the source to the destination - the newer file in the destination will be overwritten, the extra file in the destination will be purged
rem robocopy %SOURCE% %DEST% /E /COPYALL /PURGE /MT /R:1 /W:1 /V /NP /TEE /LOG+:%LOG_FILE% /L

rem /ZB option
rem Use /ZB option when file cannot be copied
rem robocopy %SOURCE% %DEST% /E /ZB /COPYALL /PURGE /MT /R:1 /W:1 /V /NP /TEE /LOG+:%LOG_FILE% /L

rem *************************************************
rem default option - changed, newer, and older file is copied; extra file is not impacted
rem Copy the source to the destination - the newer file in the destination will be overwritten, but keep the extra file in the destination
rem robocopy %SOURCE% %DEST% /E /COPYALL /MT /R:1 /W:1 /V /NP /TEE /LOG+:%LOG_FILE% /L

rem *************************************************
rem /XO option
rem Copy the source to the destination - keep the newer file and extra file in the destination
rem robocopy %SOURCE% %DEST% /E /COPYALL /MT /R:1 /W:1 /V /NP /TEE /LOG+:%LOG_FILE% /XO /L

rem *************************************************
rem /FFT option
rem Copy the source to the destination - when the third-party storage system is used
rem robocopy %SOURCE% %DEST% /E /COPYALL /MT /R:1 /W:1 /V /NP /TEE /LOG+:%LOG_FILE% /FFT /L

Robocopy Usage Summary

Robocopy is a feature-rich and powerful file migration tool in Windows. The built-in help system (robocopy /?) provides the basic usage of each option, but it doesn’t thoroughly explain why and how to use these options.

Recently I discovered a best Robocopy guide (robocopy.doc) in Windows Server 2003 Resource Kit Tools. To access this document without installing, use 7-Zip to extract the rktools.exe file, then extract the rktools.msi file. Even this document is for Version XP010, it is still applicable to the version built-in Windows 10.

I summarize some key information from this Robocopy document along with my usage experience below. It is highly recommended to read the full document if you have time.

What does Newer, Orlder or Extra mean?

File Class

In Source

In Destination

Source/Dest File Time

Source/Dest File Size

Source/Dest Attribute

Lonely

Yes

No

n/a

n/a

n/a

Tweaked

Yes

Yes

Equal

Equal

Different

Same

Yes

Yes

Equal

Equal

Equal

Changed

Yes

Yes

Equal

Different

n/a

Newer

Yes

Yes

Source > Dest

n/a

n/a

Older

Yes

Yes

Source < Dest

n/a

n/a

Extra

No

Yes

n/a

n/a

n/a

Mismatched

Yes (file)

Yes (Directory)

n/a

n/a

n/a

  • By default, Changed, Newer, and Older files are candidates for copying (subject to further filtering, as described later)
  • Same files are not copied
  • Extra and Mismatched files and directories are only reported in the output log
  • Tweaked files are neither identified nor copied - they are usually identified as Same files by default. Only when /IT is used, then Tweaked files will be copied

Use /L option to test Robocopy command

After building a Robocopy command with the options I want, I append /L option at the end of the command.

/L option lists files without copying, deleting, or applying a time stamp to any files.

Running the Robocopy command with the /L option, I can analyze the output to confirm the outcome before the real execution.

Should I use /Z, /B, or /ZB option?

  • /B – copy all files in Backup Mode. Enable you to copy some files as a Backup Operator that you would not be able to copy as a normal user. Backup mode copies are not restartable.
  • /ZB – try to copy files in Restartable Mode (for greater resiliency), and automatically switch to Backup Mode if the Restartable copy fails with an “Access Denied” error.
  • /Z, /B, or /ZB restart from close to the point of failure rather than the beginning of the files. This can decrease Robocopy performance and throughput significantly, as these options involve extra overhead. These options are only recommended when experience indicates you really need them.

My comment: not to use them unless you are not able to copy a file or the network connectivity is unstable.

Robocopy reports the file timestamp is different, even they look the same in Windows Explorer.

This usually happens when copying files between a Windows NTFS volume and a third-party storage system.

FAT file system stores its timestamp for files (modified date, etc) with 2-second granularity

NTFS file system stores its timestamp for files in a 100-nanosecond granularity

File-time granularity is 100 nanoseconds on NTFS, and two seconds on FAT. So copying a file from NTFS to FAT causes file times to be rounded to a value that file FAT file system can manage. When both the source and destination volumes are NTFS, file times are compared exactly. When either the source or destination volume is not an NTFS volume, Robocopy considers file times to be identical if they are within two seconds of each other.

Some third-party operating systems declare that their volumes are NTFS, but only implement FAT-style file times with a 2-second granularity. When copying to such a destination from a true NTFS volume, file time rounding may occur, along with unnecessary copying of file data in subsequence jobs. In this situation, use /FFT (FAT File Time) to tell Robocopy to consider file times to be identical if they are within 2 seconds of each other, even though both source and destination appear to be NTFS. This eliminates unnecessary copying in this scenario.

/FFT is useful for copying to third-party storage systems (when copying from Windows-to-Windows NTFS storage, this option is not needed) that declare a volume to be NTFS but only implement file time with a 2-second granularity.

/FFT forces Robocopy to use FAT file-time granularity so that the utility uses a 2-second standard for comparing files. As long as files are time-stamped within a 2-second interval, Robocopy considers them identical and doesn’t copy them. (Source: WindowsITPro Robocopy FAQ)

Two-way file change synchronization

/XO excludes “Older” files.

To synchronize two directory trees so that they can be updated simultaneously in separate areas (ensure that the latest files are present in both directory trees). First copy with /XO in one direction, and then in the other.

Exclude File or Directory by name

  • /XF file [file] excludes Files matching given names/paths/wildcards
  • /XD dirs [dirs] excludes Directories matching given names/paths
    • My comment: to exclude multiple directories, simply list them with spaces in between. If the path contains spaces, wrap it in double-quotes (“ ”). The most important thing is DO NOT EVER include a trailing “\” in the excluded paths.
  • /XF and /XD exclude files and directories before all other processing

Refresh security information without copying file data

  • /IS /COPY:SOU refreshes all security information for all selected files, without copying any file data

Directory fragementation for large directories

/CREATE – Robocopy creates only the directory tree and zero-length files as placeholders for the actual files. No file data is copied.

By default (without /CREATE), for the first time copy, the directory entry for each file is created first, and then the file data is copied. If the directory tree contains large directories, however, this sequence can cause fragmentation of the directory structure.

Minimize directory fragmentation for large directories

Run Robocopy with /CREATE (along with whatever the options supposed to be) first, then run the same command again without /CREATE to copy the file data to the destination.

When using this technique, the zero-length files created by the initial /CREATE command are tagged as “Changed” files on the second run.

My comment: The Robocopy version XP010 document states the files are tagged as “Older” files on the second run. When I tested on Windows 7 Pro, it shows “Changed” instead. The “Changed” matches the description in the File Classes for Changed - Source & Destination file times are equal, but file sizes are different.

Install hotfix for Windows 7 and Server 2008 R2 build-in Robocopy

Install the hotfix in KB2639043 to fix the Discretionary Access Control Lists (DACLs) for files issue.

Use WinSCP to Transfer Files in vCSA 6.7

This is a quick update on my previous post “ Use WinSCP to Transfer Files in vCSA 6.5 ”. When I try the same SFTP server setting in vCSA 6.7...