Microsoft Cluster Shared Volume Caching

I recently learned of a feature which had previously been unknown to me – Microsoft Cluster Shared Volume (CSV) caching. I use both Hyper-V and VMware at work though typically I focus on VMware as that is where the majority of our environment runs. I am deploying more systems to Hyper-V every day and the hosts all connect to CSV disks presented from our HPE 3PAR storage arrays.

CSV caching is a feature which allows a host system to use a portion of its RAM as a read cache for CSV disks. This has the advantage of caching frequently read data on the host itself negating the need to access the storage system. If we are reducing the number of read IO operations from the host we are of course taking that work off the storage which also helps. Caching is done at the block level so we don’t have to cache an entire file (for example a VHDX virtual machine disk file) which allows for granular caching of just what we need. This solution is similar to the accelerator cards you see in systems (e.g FusionIO) or where flash memory is used as a local cache. The difference here is we are using RAM instead of flash.

Obviously the most benefit will be seen in read intensive workloads however Microsoft indicate that even a minimum of 512MB of RAM allocated can provide a performance benefit. Virtual machine environments, especially VDI (Virtual Desktop Infrastructure) setups will more than likely see good results. Another example is Microsoft Scale-out File servers where large amounts of RAM can be allocated from the host.

It’s very easy to configure and use CSV caching so let’s get right to it.

 

CSV Caching Requirements and Considerations

  • Microsoft Server 2012 host operating system and above
  • Microsoft Failover Cluster with CSV disks presented
  • Available RAM on each host in the cluster
    • The RAM allocation is a cluster wide setting therefore all hosts are configured with the same value
  • Server 2012 allows up to 20% of total RAM to be allocated from the non-paged pool
  • Server 2012 R2 allows up to 80% of total RAM to be allocated from the non-paged pool
  • Microsoft currently recommend you DO NOT exceed a cache size of 64GB

Note the following considerations from Microsoft (http://blogs.msdn.com/b/clustering/archive/2012/03/22/10286676.aspx) –

For the Hyper-V root memory reserve in the parent partition to be modified to accommodate the memory allocated to the CSV cache it does require a server reboot with Windows Server 2012.  To ensure resource contention is avoided, it is recommended to reboot each node in the cluster after modifying the memory allocated to the CSV cache.  On Windows Server 2012 R2 the Hyper-V room memory reserve will be dynamically updated and a reboot is not necessary.
Enabling CSV Cache on an individual disk requires that the Physical Disk resource be recycled (taken Offline / Online) for it to take effect.

CSV Cache will be disabled on:

  • Tiered Storage Space with heat map tracking enabled
  • Deduplicated files using in-box Windows Server Data Deduplication feature (Note:  Data will instead be cached by the dedup cache)
  • ReFS volume with integrity streams enabled (Note:  NTFS is the recommended file system as the backend for virtual machine VHDs in production deployments)

 

Monitoring CSV cache metrics is done via Perfmon.exe or Typeperf.exe depending on your setup. It is important to note that the Perfmon counters do not become active on a host until a reboot has been completed. The CSV cache is live and working but those counters will register nothing till after the reboot so it can be misleading.

 

CSV Cache Configuration Server 2012 R2

Enable CSV Cache

If you are using Server 2012 R2 then the option should be enabled by default, we can check with the PowerShell command below (executed locally on a cluster node) –

PS C:\> Get-ClusterSharedVolume | Get-ClusterParameter | where {$_.Name -eq "EnableBlockCache"}

 

Object              Name                Value               Type
------              ----                -----               ----
GP_bsa_hv_vv01      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv02      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv03      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv04      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv05      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv06      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv07      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv08      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv09      EnableBlockCache    1                   UInt32
GP_bsa_hv_vv10      EnableBlockCache    1                   UInt32
GP_bsa_hvgponly_vv  EnableBlockCache    1                   UInt32
GP_bsa_lyncbe01_vv  EnableBlockCache    1                   UInt32
GP_bsa_sccm_vv      EnableBlockCache    1                   UInt32
GP_bsa_scom_vv      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv11      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv12      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv13      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv14      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv15      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv16      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv17      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv18      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv19      EnableBlockCache    1                   UInt32
SP_bsa_hv_vv20      EnableBlockCache    1                   UInt32
SP_bsa_hvsponly_vv  EnableBlockCache    1                   UInt32
SP_bsa_lyncbe02_vv  EnableBlockCache    1                   UInt32
SP_bsa_scsm_vv      EnableBlockCache    1                   UInt32
SP_bsa_vmm_vv       EnableBlockCache    1                   UInt32

 

You will see the property ‘EnableBlockCache’ is set to 1 which means it is enabled.

If you want to execute the PowerShell command remotely then the below is one way of doing so –

get-cluster -Name "BSA-HVCL" | Get-ClusterSharedVolume | Get-ClusterParameter | where {$_.Name -eq "EnableBlockCache"}

 

Define Cache Size

Now that we have enabled/checked CSV caching is turned on it’s time to set the amount of RAM we want to allocate on each of the cluster nodes.

To begin with let us check what the existing setting is from a cluster node PowerShell prompt –

PS C:\> (get-cluster).blockcachesize
0
PS C:\>

This indicates that no RAM is allocated for caching. If we want to check this setting from a remote system we could use the following PowerShell command –

PS C:\> (get-cluster -Name "BSA-HVCL").blockcachesize
0
PS C:\>

We could of course create a variable for the cluster –

PS C:\> $cluster = get-cluster -Name "BSA-HVCL"
PS C:\> $cluster

Name
----
BSA-HVCL

PS C:\> $cluster.blockcachesize
0
PS C:\>

 

Now that we have checked our existing setting it’s time to define how much RAM to allocate. The value is entered in megabytes, in my case I will set 1024MB as a RAM cache.

PS C:\> (get-cluster -Name "BSA-HVCL").blockcachesize

Or using our variable example -

PS C:\> $cluster.blockcachesize = 1024

If we now check the option again we will see the new value –

PS C:\> $cluster.blockcachesize
1024
PS C:\>

 

If you wish to disable the CSV cache the best way is to change the cache size to 0 rather than disabling the ‘EnableBlockCache’ property.

 

CSV Cache Configuration Server 2012

Enable CSV Cache

If you are running Server 2012 firstly enable CSV caching on the CSV disk(s). For example, let us enable the CSV cache option for a single CSV disk. The command below was executed locally on a cluster node –

PS C:\> Get-ClusterSharedVolume “Name of CSV Here” | Set-ClusterParameter CsvEnableBlockCache 1

Define Cache Size

Server 2012 requires a slightly different command to define the cache size as the property was renamed for 2012 R2.

PS C:\> (Get-Cluster). SharedVolumeBlockCacheSizeInMB = 1024


 

Hopefully this post has been useful – if you have any questions or suggestions please feel free to leave a comment.

5 thoughts on “Microsoft Cluster Shared Volume Caching”

  1. I have enable CSV cache on my cluster and made sure it was running, When I add the counters in Perfmon.exe called Cluster CSV Volume Cache to check how much is being used, I see no activity. I am running server 2012r2 on 8x RDVH servers. I can log into each and run the commands and see that it is running on all volume. Do you know any way for me to check to make sure this feature is actually being used?

    Reply
  2. I allocated 25GB or 25600MB to the cache, Here is my output.
    PS C:\Users\administrator> Get-ClusterSharedVolume | Get-ClusterParameter | where {$_.Name -eq “EnableB
    ockCache”}

    Object Name Value Type
    —— —- —– —-
    SSD_Vol1 EnableBlockCache 1 UInt32
    SSD_Vol2 EnableBlockCache 1 UInt32
    SSD_Vol3 EnableBlockCache 1 UInt32

    PS C:\Users\administrator> (get-cluster).blockcachesize
    25600
    PS C:\Users\administrator>

    Reply
  3. I applied 25600MB

    PS C:\Users\> Get-ClusterSharedVolume | Get-ClusterParameter | where {$_.Name -eq “EnableB
    ockCache”}

    Object Name Value Type
    —— —- —– —-
    SSD_Vol1 EnableBlockCache 1 UInt32
    SSD_Vol2 EnableBlockCache 1 UInt32
    SSD_Vol3 EnableBlockCache 1 UInt32

    PS C:\Users\> (get-cluster).blockcachesize
    25600

    Reply
  4. Windows PowerShell
    Copyright (C) 2016 Microsoft Corporation. All rights reserved.

    PS C:\MUNIT> Get-ClusterSharedVolume | Get-ClusterParameter | where {$_.Name -eq “EnableBl
    ockCache”}

    Object Name Value Type
    —— —- —– —-
    SSD_Vol1 EnableBlockCache 1 UInt32
    SSD_Vol2 EnableBlockCache 1 UInt32
    SSD_Vol3 EnableBlockCache 1 UInt32

    PS C:\MUNIT> (get-cluster).blockcachesize
    25600

    Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.