I’m currently working on a case with HPE (Hewlett Packard Enterprise) 3PAR support with regard to excessive retransmit errors on our Remote Copy Fibre Channel (RCFC) connection between two 7400 arrays. While troubleshooting we found that a required option was not set appropriately – namely interrupt coalescing on the RCFC ports. In this post I will cover how to change the setting and in brief what it actually means.
Interrupt Coalescing – what is it?
First off let us discuss what interrupt coalescing is and how it relates to the 3PAR remote copy ports.
Without delving too deeply into the topic we can simply say that interrupt coalescing is the process by which I/O that would normally trigger an interrupt/context switch to the CPU is gathered together (coalesced) until we either have a certain number of I/O’s or we hit a time value. To give an example we might have a system which is configured to wait for 10 I/O’s to coalesce before interrupting the CPU. Alternatively it might be set to coalesce I/O’s for 2 milliseconds and then send the interrupt.
The idea of interrupt coalescing is to reduce the interrupt/context switching load on processors and in the right scenarios it can be beneficial.
If we consider the HPE 3PAR array specifically and RCFC then the following is accurate to my knowledge –
- Interrupt coalescence is a way of reducing the number of context switches that the CPU makes when processing events from the fibre channel HBA (Host Bus Adapter)
- If you enable coalescence you are telling the HBA to delay the generation of interrupts until either:
- Two interrupt events have arrived
OR - 1ms has elapsed since an interrupt event has arrived
- Two interrupt events have arrived
- If the I/O profile is multithreaded and generates lots of concurrent I/O then most of the time condition 1 will apply
- If you have a single threaded application that sends one I/O at a time you will fall into condition 2
In this scenario we have two HPE 3PAR 7400 arrays which are 20 miles apart using Remote Copy Fibre Channel (RCFC) to synchronously replicate to one another. Below is a screenshot from the management console which shows the connectivity – we have two 8Gb ports for replication traffic.
Configure Interrupt Coalescing on HPE 3PAR
First let us check which ports have interrupt coalescing (or IntCoal as the 3PAR calls it) configured. To do this on the command line we can make use of the showport command with the –par switch.
BSA-3PAR01 cli% showport -par N:S:P Connmode ConnType CfgRate MaxRate Class2 UniqNodeWwn VCN IntCoal TMWO Smart_SAN 0:0:1 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 0:0:2 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 0:1:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:1:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:3 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:4 rcfc point auto 8Gbps disabled disabled disabled enabled disabled unlicensed 1:0:1 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 1:0:2 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 1:1:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:1:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:3 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:4 rcfc point auto 8Gbps disabled disabled disabled enabled disabled unlicensed --------------------------------------------------------------------------------------------------
We can also check via the GUI, both through the traditional management console (IMC) and the newer web StoreServ Management Console (SSMC) –
As you can see the RCFC port in question has IntCoal enabled – it should be noted it is enabled by default so it is up to the storage administrator to know to disable it.
The steps involved are as follows –
- Stop Remote Copy (RC) groups
- Changing IntCoal settings can result in the port going offline for a short time (seconds) therefore it is safer to stop remote copy synchronisation while making the changes
- Change IntCoal setting via the command line
- Start Remote Copy groups
As you can see the process is very simple, it’s just important to make sure you stop your RC groups before and be aware the port going offline will trigger an alert.
We leverage the controlport command as shown below –
BSA-3PAR01 cli% controlport intcoal disable 0:2:4 Are you sure you want to run controlport intcoal disable on port 0:2:4? select q=quit y=yes n=no: y BSA-3PAR01 cli% controlport intcoal disable 1:2:4 Are you sure you want to run controlport intcoal disable on port 1:2:4? select q=quit y=yes n=no: y BSA-3PAR01 cli%
As you can see I ran the command twice to configure both ports – please note you need to configure this on all RCFC ports involved, i.e if you have two arrays the setting must be configured on both.
If we now check the settings on one of the arrays we can see the RCFC ports now have IntCoal disabled. Again we leverage the showport command with the –par switch –
BSA-3PAR01 cli% showport -par N:S:P Connmode ConnType CfgRate MaxRate Class2 UniqNodeWwn VCN IntCoal TMWO Smart_SAN 0:0:1 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 0:0:2 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 0:1:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:1:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:3 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 0:2:4 rcfc point auto 8Gbps disabled disabled disabled disabled disabled unlicensed 1:0:1 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 1:0:2 disk point 6Gbps 6Gbps n/a n/a n/a enabled n/a unlicensed 1:1:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:1:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:1 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:2 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:3 host point auto 8Gbps disabled disabled disabled disabled enabled unlicensed 1:2:4 rcfc point auto 8Gbps disabled disabled disabled disabled disabled unlicensed --------------------------------------------------------------------------------------------------
I hope the information in this post has been useful – if you have any thoughts or questions please drop a comment below.