Trend Micro Deep Security Filter Driver Heap Size

Trend Micro Deep Security Filter Driver Heap Size

My VMware environment is protected by Trend Micro Deep Security to provide agentless protection of virtual machines (VMs). Recently I encountered a ‘Purple Screen of Death’ (PSOD) on one of the ESXi hosts which appeared to be caused by Deep Security.

Further investigation indicated that the filter driver heap size had not been modified. Deep Security primarily uses the filter driver heap to maintain connection state tables and configuration data. If the allocated maximum size is not sufficient it can cause various problems. It is also necessary to ensure the Deep Security Virtual Appliance (DSVA) has sufficient vCPU and RAM allocated.

Having identified the issue it was time to modify the settings on our ESXi hosts, this is a relatively simple process which I will outline below.

For reference, Trend Micro provide an FAQ which covers this topic – http://esupport.trendmicro.com/solution/en-US/1095995.aspx

Determine Heap Size Requirement

The first thing we need to do is determine how much memory should be allocated to our heap size. Based on the Trend Micro FAQ we can use the following tables as a guide, in my case the Deep Security 9.5 table is appropriate.

Deep Security 9.5

Number of VM Guests	1 – 30	31 – 90	91 – 100
DSVA Memory	4GB	6GB	8GB
DSVA CPUs	2 x vCPU	2 x vCPU	2 x vCPU
Dv_filter Heap Size	256MB	1GB	1GB

Deep Security 9.0 and 8.0

Number of VM Guests	1 – 50	100	150	200	250
DSVA Memory	2GB	4GB	8GB	8GB	12GB
DSVA CPUs	2 x vCPU	2 x vCPU	4 x vCPU	4 x vCPU	6 x vCPU
Dv_filter Heap Size	256MB	512MB	1GB	1GB	1.5GB

We can see various configurations dependent on version and load – I need to set my environment with the following –

Number of VM Guests	31 – 90
DSVA Memory	6GB
DSVA CPUs	2 x vCPU
Dv_filter Heap Size	1GB

Modify Heap Size and DSVA Settings

Now that I have my settings it’s time to configure everything. To start with I shutdown the DSVA appliance on each host and increased the RAM allocation to 6GB. I then connected to each ESXi host in turn and ran the necessary commands to increase the filter driver heap memory size.

To check whether the heap size has been modified before we can use the following command –

esxcfg-module -g dvfilter-dsa

If the value has not been modified before we would see the following –

dvfilter-dsa enabled = 1 options = ''

If the value has been altered we would see something similar to the following –

dvfilter-dsa enabled = 1 options = 'DSAFILTER_HEAP_MAX_SIZE=1074790400'

In my case the value had not been modified before. I ran the following command to alter the maximum heap size. Please note, Trend Micro indicate a value in bytes which isn’t exactly 1GB. I spoke with their support team as the KB article gives an example of a heap size of 512MB and the value is 512000000. In my mind this should mean that a 1GB value would be 1024000000 and not the 1074790400′ they provide. The support engineer indicated that the 1GB value they provide in the KB should be followed therefore I will use this value in the configuration.

~ # esxcfg-module -s "DSAFILTER_HEAP_MAX_SIZE=1074790400" dvfilter-dsa

~ # esxcfg-module -g dvfilter-dsa

dvfilter-dsa enabled = 1 options = 'DSAFILTER_HEAP_MAX_SIZE=1074790400'

We can see that the output now shows the modified heap size. At this point it is necessary to reboot the ESXi host so put it into maintenance mode and shift all the virtual machines off then reboot. Once the host has started and shows in vCenter again you can bring it out of maintenance mode and allow DRS to move VMs back.

I would strongly recommend reading the Trend Micro FAQ linked above to ensure you understand all of the options and requirements before making a change on production systems.