As most Unidesk customers know when we create layers we capture changes to the file system at a block level. This means that deleted files for the most part take up the same room as regular files. Normally this is not that big a deal as the percentage of deleted files to the overall number of files is normally pretty small.
Recently Microsoft has changed its method of delivering what used to be called service packs and they now do a major upgrade as an in-place upgrade to Windows about once a year. These upgrades expand the size of the OS layer by a very large amount. I took a look at a Windows 10 OS layer I had created with version 1511 and it used about 12 GB of space. Over time with updates that had grown to about 20 GB of space. I then performed the 1511 to 1607 Anniversary Edition upgrade and my OS layer was now 38 GB.
That was enough to get me to finally work out how to compact a VMDK and remove all of the deleted blocks on the disk.
Luckily this is not a hard process though you do need to have root access to at least one ESX host to perform it.
The first tool I had to find was one that would zero out the already deleted blocks. Since I am working on Windows, I searched for that and found that Mark Russinovich had written such a utility called SDelete. To obtain the utility just google it and get the latest version.
The process is pretty simple:
- First ensure that you have enough free disk space for a fully expanded layer on the datastore where your Unidesk MCP resides. If you are doing an OS layer you will need a bit more than 100 GB. For a standard App layer you will need 10 GB of free space.
- Add a new version to your OS layer (or any other layer you want to compress).
- Log on to the Install Machine and copy down the SDelete to the machine. I created a folder off the root called tools and put it there. I also deleted it after running it.
- Run sdelete –z c:
This will run for quite a while. It will expand the VMDK of the layer to the full size of the disk as it zeros out the deleted blocks. For me the process went to 100% but did not exit. It didn’t hurt anything I just rebooted.
- Then shut down the Install Machine. Do not finalize yet.
- Then SSH to an ESX host with access to the datastore the Unidesk MCP layers are on. Log on as root.
- Change to the directory where the OS layer is located. This should be:
/vmfs/volumes/<datastore name> /<mcp name> /UnideskLayers/OS/<OS Layer Name>
- Then run the vmkfstools command to remove zeroed out blocks:
Vmkfstools –-punchzero <vmdk filename>
As that runs if you use the datastore browser you will see the command shrinking the VMDK.
- When complete you can go and finalize the layer.
That’s it. It's actually a pretty easy process. To give you an idea of how well it works my original Windows 10 layer was 38 GB and it shrunk down to 15 GB using this process.
For Hyper-V go here
My original blog ended right here. Then my boss challenged me to automate the process. So I created a new utility that allows you to choose an install machine and it will compress the layer then finalize it. The idea here is that you remove the deleted space for every layer before finalizing.
I created the utility which you can find here http://www.unidesk.com/forum/scripts-and-utilities/unidesk-2x-layer-compression-utility using an HTA, PowerShell scripting and plink. The utility works a little differently than the manual process as I decided it would be safer and it would likely work better to mount the layer on a separate console machine to zero out the deleted blocks. That way you don’t have to deal with the Install Machine running out of free space to write to the disk. The side benefit was that I could run my utility on the same machine and therefore have access to the disk. I also decided not to use SDelete but to instead use part of the script created here by Chris Duck: http://blog.whatsupduck.net/2012/03/powershell-alternative-to-sdelete.html
This is a neat idea, basically creating a file that fills all the free space on the disk with zeros.
When you run the utility it queries the Management Appliance database and lists all current layer Installs that are running. You select the desired layer and click Compress Layer.
This runs a script that will:
- Shut down the installation machine
- Mount the VMDK on the console machine
- Fill the deleted and unused space with zeros
- Run the vmkfstools command to zero out deleted blocks in the VMDK
- Unmount the VMDK on the console machine
- Start up the Install Machine so it re-associates itself with the disk
- Then finalize the layer via web call to the MA
I like how it came out. It's pretty easy to use and handy.
After I published the original blog for vSphere, someone asked about HyperV. In HyperV, the idea is the same but the process is different.
- First create a new version of your layer
- On a GUI HyperV Server log in as an Administrator
- Shut down the Install Machine
- Find the layer in the MCP storage on your Server.
- Should be a path something like:
- Right click on the VHD and mount it
- If it doesn’t get a drive letter open disk manager and assign it one
- Open an Admin CMD prompt
- Go to the location where you copied sDelete64
- Run sdelete64 –z d: (or whatever drive you mounted)
- When SDelete gets to 100% you can close the cmd prompt even though it's still running
- Right click on the mounted disk and eject it
- Open an Admin CMD prompt again and run DISKPART
- In DISKPART First run Select vdisk File="<file path>"
- Then compact vDisk
- Then run detach vDisk
- Then start the Install Machine
- And lastly finalize the layer
I will add a task to my list to also create a utility to do this easier.