I had to troubleshoot a Microsoft Failover Cluster with a SQL instance recently and it proved an interesting one. This cluster uses the HP 3PAR Cluster Extension software (CLX) to interact with the 3PAR arrays in our datacenters and facilitate the movement of the SQL instance between sites.
Note – the HP 3PAR CLI (command line) interface software also needs to be installed on nodes, however in this post I will focus on the CLX side of things.
For some reason the instance could no longer move successfully between the nodes in this cluster. In fact it would attempt to move, take all the cluster resources offline and then fail to bring them online. I took a look at the arrays to make sure all hosts were still zoned in and had active paths logged in and everything looked good. I did however note that in my task list no entries for a remote copy reverse existed which was strange.
I decided to move back to the cluster nodes and check the event logs for any sign of issue and behold I found the following –
Log Name: Application Source: 3PARClxEvents Date: 05/05/2015 13:17:41 Event ID: 999 Task Category: (1) Level: Error Keywords: Classic User: N/A Computer: SQLFOC03N01.bytesizedalex.com Description: Dumping AutoPass ERR obj:\n Error Number = 5997\n Error Source = COvLicReporter::OvLicReportLicense\n Error Message = No Passwords in License File\n Event Xml: <Event xmlns="<a href="http://schemas.microsoft.com/win/2004/08/events/event"">http://schemas.microsoft.com/win/2004/08/events/event"</a>> <System> <Provider Name="3PARClxEvents" /> <EventID Qualifiers="0">999</EventID> <Level>2</Level> <Task>1</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-05-05T12:17:41.000000000Z" /> <EventRecordID>25509</EventRecordID> <Channel>Application</Channel> <Computer>SQLFOC03N01.bytesizedalex.com</Computer> <Security /> </System> <EventData> <Data>Dumping AutoPass ERR obj:\n Error Number = 5997\n Error Source = COvLicReporter::OvLicReportLicense\n Error Message = No Passwords in License File\n</Data> </EventData> </Event>
The section that caught my attention was ‘Message = No Passwords in License File\n’. I knew that this cluster had worked as full failover testing had been carried out so why was it no longer working?
HP CLX software creates a password file for each array it connects to, typically in c:\HP3PARCLIPassword
I have two 3PAR 7400 arrays, one per datacenter therefore this cluster node has two password files. inside each file is a single line entry with the username followed by a password.
So we have our password files and we have a username and password in them so what gives? Well if we take another look at the error message it also talks about a license file and this is the key to fixing our problem.
At this point I had tested the connection between nodes and arrays without any trouble so I knew the username/password was OK, the next step was to try licensing my nodes again.
This is a simple process, note you will need your license files.
- Open a Command prompt on any cluster node member (Run as Administrator)
- Make sure the account you are logged in with has local admin rights on all cluster nodes, typically a service account is used for CLX work so I would suggest logging on with this account
- Change directory to the CLX bin folder – default is C:\Program Files\Hewlett-Packard\Cluster Extension 3PAR\bin
- Run ClxDeviceLicenseRegistration.exe and provide the path to your license files, you can enter more than one file to license multiple arrays
- Example –
C:\Program Files\Hewlett-Packard\Cluster Extension 3PAR\bin>ClxDeviceLicenseRegistration.exe -f c:\hp\clx\1234567\AB123AAA_1234567.dat c:\hp\clx\7654321\AB123AAB_7654321.dat This tool should only be used to apply device based permanent license. This tool should NOT be used for applying Node based permanent license. Once the valid device based permanent license files are supplied, this tool would import the licenses not only in the current cluster node, but also would import to all the nodes of this cluster. Do you want to continue? [Y] Yes [N] NO (Default is N) : Y c:\hp\clx\1234567\AB123AAA_1234567.dat : Valid license file c:\hp\clx\7654321\AB123AAB_7654321.dat : Valid license file Applying license :: Node 1 : SQLFOC03N02 : Success Node 2 : SQLFOC03N01 : Success Node 3 : SQLFOC03N03 : Success Node 4 : SQLFOC03N04 : Success C:\Program Files\Hewlett-Packard\Cluster Extension 3PAR\bin>
You will notice the system licenses all nodes in the cluster which is a really useful feature. After this task completed I tried to migrate the role to another node and (huzzah) it worked. The resource moved as expected and everything came online in failover cluster manager.