HP 3PAR Quorum Witness Server Deployment and Configuration

by

In my environment we have two HP 3PAR 7400 arrays configured for synchronous replication. They are separated by about 16 miles with a 10Gb DWDM (dense wave devision multiplexing) link. We use Peer Persistence for VMware and HP CLX cluster extensions for Hyper-V to enable seamless failover between arrays.

A vital component is the HP 3PAR quorum witness server which arbitrates in a split-brain scenario. Today we shall look at the deployment and configuration of the 3PAR quorum witness server appliance.

Note: While I will try to cover the process in detail I am assuming a basic understanding of the 3PAR quorum witness design and purpose.

 

What is a Quorum Witness?

The HP 3PAR Quorum Witness is a virtual machine (VM) appliance deployed on ESXi that acts as an arbitrator in a split-brain scenario. You should always deploy the witness server on a third site with diverse routes to your two datacentres (DCs). As the witness is deployed to arbitrate in a split-brain scenario it needs to be hosted externally to your DCs.

The system essentially works by polling each 3PAR and allowing them to communicate back to the witness. In the event of a failure the witness will aid the decision making process of whether a 3PAR array should cease control of any volumes exported by it’s partner.

For a high level overview please see my post – https://www.bytesizedalex.com/hp-3par-quorum-witness-role/

 

3PAR Quorum Witness OVF Deployment

I’m going to assume you have downloaded the latest copy of the 3PAR quorum witness appliance and have access to VMware vCenter to import the OVF.

There are a number of ways to import the appliance, the screenshots below will demonstrate one method.

Logon to the vSphere web client and use the menu option to deploy and OVF template.

Import OVF Step 1

Click ‘Next’ and then ‘Browse’.

Browse to OVF Location

Select the downloaded OVF file.

Select OVF File

Click ‘Next’ to continue.

OVF Selected Click Next to Continue

Review the summary for the quorum witness appliance then click ‘Next’ to continue.

Review OVF Summary

Enter a suitable name for the virtual machine, select a location in your vCenter inventory and click ‘Next’ to continue.

Enter VM name and select folder

Select a vCenter cluster/host for the virtual machine to run on – remember this should be on a third site and not on one of the two datacentres hosting your 3PAR arrays. Click ‘Next’ to continue.

Select vCenter Cluster and Host

Select the datastore for the VM and choose whether between ‘Thick’ and ‘Thin’ provisioning. I have chosen to use ‘Thin’ provisioning which is recommended. If you choose to thick provision the VM you will find it uses 256GB of space, the vast majority of which will never actually be used by the VM. Once complete click ‘Next’ to continue.

Select Datastore and Thick or Thin Provisioning

Select the correct network settings for your environment then click ‘Next’ to continue.

Select VM Network Settings

Review the deployment summary and click ‘Finish’ to deploy the OVF.

Review Deployment Summary

 

At this point the OVF file will be imported into vCenter and deployed using the settings defined during the wizard. Once deployed we can start the 3PAR quorum witness configuration.

 

3PAR Quorum Witness Configuration

Power on the VM and connect to the console. Allow the VM time to start then select your keyboard from the list available. Use the ‘TAB’ button on your keyboard to move between options. Tab to the ‘OK’ button and press the space bar or ‘Enter’ to continue.

Select Keyboard

The wizard will prompt you to enter a new password for the ‘root’ account.

Enter new Root Password

Select ‘Device Configuration’.

Select Device Configuration

Select ‘eth0’ and press Enter.

Select eth0 network adapter

Configure appropriate IP settings for the VM. I recommend you also configure your DNS server with the relevant record for this VM at this time. Once complete select the ‘OK’ button to return to the previous menu.

Configure IP Address settings

Select ‘Save’.

Select Save

You will return to a previous menu, this time select ‘DNS Configuration’.

Select DNS Configuration

Enter the required details to ensure hostname, DNS servers and DNS search path are complete then select ‘OK’.

Configure DNS Settings

Select ‘Save&Quit’.

Select Save & Quit

You will be presented with a new menu. Do not make any changes, select the ‘Cancel’ option.

Select Cancel

You will be presented with a new menu. Do not make any changes, select the ‘Cancel’ option.

Select Cancel

The system will now bring up the network interface and complete configuration.

Wizard activates network

Once you are presented with a logon prompt authenticate as the root user with the password defined earlier.

Logon as root

We need to make some changes to a configuration file. Use the command below –

vi /etc/sysconfig/network-scripts/ifcfg-eth0

 

The command will open the configuration file –  you need to comment out or delete the ‘PREFIX’ line (e.g ### PREFIX=20).

In my deployment this line did not exist therefore I exited vi. If you do find the ‘PREFIX’ line exists then comment it or delete then save the file and exit vi.

Review ifcfg-eth0 file

Now verify the IP address information for the VM is correct using the ‘ifconfig’ command.

Verify IP settings with ifconfig

We will verify that the quorum witness appliance is functioning correctly. Enter the command ‘curl http://[serverNameFQDN]:8080’

For example –

curl http://bsa.3parqw01.bsa.com:8080

The expected response is –

{"couchdb":"Welcome","version":"1.0.4"}

You can also test functionality by using ‘localhost’ as a hostname however you should always try using the system FQDN to properly validate.

Verify site connectivity curl http://[serverNameFQDN]:8080

Now we will test the witness database – here I have demonstrated using ‘localhost’ though again you should really check this using the FQDN. Enter the command ‘curl http://[serverNameFQDN]:8080’/witness’

For example –

curl http://bsa.3parqw01.bsa.com:8080/witness

The return should be similar to the following example. The actual values may vary, but doc_count should be at least 1 and the not_found error should not occur.

{"db_name":"witness","doc_count":1,"doc_del_count":0,"update_seq":1,"purge_seq":0,"compact_running":false,"disk_size":4185,"instance_start_time":"1407454201619825","disk_format_version":5,"committed_update_seq":1}

Verify Witness Database curl http://localhost:8080/witness

 

Assuming no errors have occurred the 3PAR quorum witness has been successfully deployed and configured. At this point we could configure our 3PAR arrays to use the witness however I would suggest ensuring the VMware tools components are updated before proceeding. Mount the VMware tools on the VM via vCenter then use the commands below to install using the default options.

mount /dev/cdrom /mnt/cdrom
cp /mnt/cdrom/VMwareTools-*.tar.gz /tmp
tar -zxf /tmp/VMwareTools-*.tar.gz -C /tmp
cd /
./tmp/vmware-tools-distrib/vmware-install.pl --default

Cleanup the remaining extracted files –

rm -f /tmp/VMwareTools-*.tar.gz
rm -rf /tmp/vmware-tools-distrib

Hopefully I’ve covered everything in sufficient detail however if you do have any questions please feel free to comment below and send me a Tweet so I can assist.

In my next post we will look at completing the witness configuration on our 3PAR arrays.

8 thoughts on “HP 3PAR Quorum Witness Server Deployment and Configuration”

  1. Hello,

    Thanks for this great post.
    is there a way to verify that 3par array can communicate with the witness ? I mean, is there any 3par witness service that I can monitor or a command that verify the connection with 3par witness ?

    Regards

    Reply

    • Hi EMD,

      Thanks for the positive feedback. The quick answer is yes you can check via the 3PAR management software (traditional console, web or CLI) to see if there is witness communication. IF you want to monitor from something else it depends on the tools you have available to you. For example, I use the SolarWinds monitoring tools in my business to check the URL and report back to me if it doesn’t respond. I’m looking at enhancing that to look for specific strings in the returned data so I can be sure it really is working. Alternatively you could use PowerShell or some other command language script to do something similar.

      As an example with PowerShell we can use the invoke-webrequest cmdlet –

      Invoke-WebRequest -Uri http://bsa-3parqw01.bytesizedalex.com:8080/witness

      If you run this substituting your witness FQDN you will get some good data back in the command response which you could then use to generate alerts. I think the best property to use would be ‘Content’. For example I know in the past when the witness has stopped working the curl command run locally would output –

      [root@BSA-3PARQW01 /]# curl http://localhost:8080/witness
      curl: (7) couldn’t connect to host

      With that in mind we could write a script using invoke-webrequest which looked at the content property for that statement and then generate an alert. I’d be really interested to hear how you go about implementing your monitoring – if you have any questions or want to discuss further let me know. Hopefully my comments have made some sense!

      Regards,

      Alex

      Reply

      • Hi Alex,
        Many thanks for the prompt reply. Using powershell is smart idea. Personally, I might use the ‘Status Code’ property if the returned value other than 200 alert is generated.
        You mentioned that I can check the witness communication via HP management software, I appreciate if you can tell me how.

        Regards,

        Reply

        • Hi EMD,

          That sounds like a good idea. With regard to the management software, it’s pretty basic – essentially you can just see whether the witness is running and the array is talking with it. With the traditional management console you can check by going to the ‘Remote Copy’ tab and viewing the ‘Heartbeat Round Trip’ chart. Also if you click on the Peer Persistence tab you can see if there is a tick in the box for both arrays. When the witness hasn’t worked this tick usually vanishes.

          If you are using the newer web client (SSMC) you can view the same information under the ‘Remote Copy Configuration’ tab by clicking on the ‘Targets’ dropdown. Again it’s pretty basic, just tells you it’s started.

          Finally you can see this info from the command line by running ‘showrcopy -qw’. This can generate a lot of text depending on your config but the bit you are interested in comes in the first few lines. As above it will just tell you if the witness is started or not.

          Off the top of my head I can’t think of any other areas to look in the 3PAR tools, they are pretty basic in what they tell you which is why I’ve started to lean towards some other way of monitoring the witness. Hopefully this all made sense, if not let me know.

          Regards

          Alex

          Reply

    • Hi Amit, I don’t have access to that 3PAR setup now I’m in a new job. The standard HPE instructions should be sufficient. I created this post as at the time the documentation for this task wasn’t great. If I get access to another setup I can try and document.

      Reply

  3. It was a great post , but iam looking for reconfiguration of quorom witness
    1) Stop QW
    2) Remove QW
    3) Configure QW back and start (IP is the same:)

    can you please help

    Reply

    • Hi Raj,

      I’m not sure what you’re asking but if you follow those steps you can easily swap a QW for a new one. It will not have an impact on production systems though of course if there was a problem which required the witness to arbitrate you’d have a situation to take care of.

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.