HP 3PAR Cage Degraded and the Upgradecage Command

I came across a fault yesterday with one of the 3PAR arrays under my management. Logging onto the SAN with the management console I discovered one of the cages showing as degraded as shown in the screenshot below –

Cage-Degraded.png

As you can see cage0 is unable to report on the firmware status of either card and therefore reports a degraded state. While the cage shows as degraded everything was working as expected so we didn’t have a service impact.

Resolution

First I connected to the array via SSH and ran the showcage command, appending the cage name to the end to filter my results to the degraded unit.

 3PAR01 cli% showcage -d cage0
 Id Name  LoopA Pos.A LoopB Pos.B Drives Temp  RevA RevB Model Side
 0 cage0 1:0:1     0 0:0:1     0     14 14-29 3211 3211 DCN1  n/a
-----------Cage detail info for cage0 ---------
Position: ---
Interface Board Info            Card0            Card1
 Firmware_status          Unknown          Unknown
 Product_Rev             3211             3211
 State(self,partner)            OK,OK            OK,OK
 VendorId,ProductId          HP,DCN1          HP,DCN1
 Master_CPU              Yes               No
 SAS_Addr 50050CC10230567E 50050CC10230567E
 Link_Speed(DP1,DP2)  6.0Gbps,6.0Gbps  6.0Gbps,6.0Gbps
PS PSState ACState DCState Fan State Fan0_Speed Fan1_Speed
 ps0      OK      OK      OK        OK        Low        Low
 ps1      OK      OK      OK        OK        Low        Low

[additional command output removed]

 

We can see that the firmware status shows as Unknown however the rest of the output is normal.

The next step involved running the command upgradecage which downloads the latest compatible firmware from the array to the designated cage(s). The command output is shown below –

3PAR01 cli% upgradecage cage0 
Upgrading cage cage0 cpuA from rev 3211 to revision in file /opt/tpd/fw/cage/ebod/hp_eos_canister_local_combined_v3.2.0.17.gff. 
Upgrading cage cage0 cpuB from rev 3211 to revision in file /opt/tpd/fw/cage/ebod/hp_eos_canister_local_combined_v3.2.0.17.gff. 
Beginning test after upgrade for cage0 
cage0 passed test after upgrade

As you can see the command completed successfully and the firmware was installed. With this step complete we now run the showcage command again to check the output.

3PAR01 cli% showcage -d cage0
 Id Name  LoopA Pos.A LoopB Pos.B Drives Temp  RevA RevB Model Side
 0 cage0 1:0:1     0 0:0:1     0     14 13-25 3211 3211 DCN1  n/a
-----------Cage detail info for cage0 ---------
Position: ---
Interface Board Info            Card0            Card1
 Firmware_status          Current          Current
 Product_Rev             3211             3211
 State(self,partner)            OK,OK            OK,OK
 VendorId,ProductId          HP,DCN1          HP,DCN1
 Master_CPU              Yes               No
 SAS_Addr 50050CC10230567E 50050CC10230567E
 Link_Speed(DP1,DP2)  6.0Gbps,6.0Gbps  6.0Gbps,6.0Gbps
PS PSState ACState DCState Fan State Fan0_Speed Fan1_Speed
 ps0      OK      OK      OK        OK        Low        Low
 ps1      OK      OK      OK        OK        Low        Low

[additional command output removed]

The firmware now shows as current and the degraded state is no longer displayed within the management console as per the screenshot below –

Cage-Normal.png

This was my first use of the upgradecage command and I’m glad to say it went smoothly and without issue.

The full help for this command is detailed below –

 

DESCRIPTION
The upgradecage command downloads new firmware into the specified cage.


SYNTAX
The syntax for the upgradecage command can be one of the following:
• upgradecage [options <arg>] <cagename>...
• upgradecage [options <arg>] –a


AUTHORITY
• Super, Service
• Any role granted the cage_upgrade right.


OPTIONS
–a
Specifies that all drive cages are upgraded with new firmware (<cagename> should not be
specified).
–noA
Do not upgrade the A loop. This option is only valid for DC1 cages.
–noB
Do not upgrade the B loop. This option is only valid for DC1 cages.
-skiptest
Skips the 10 second per PD diagnostic test normally completed after each cage upgrade.
–model <model>
Only upgrade cages of specified model. Use showcage command to find out the correct model
for the cages. The model can be DC1, DC2, DC3, or DC4.
–file <fpname>
Use firmware in file <fpname>, where <fpname> is the full path name of the file on the node.
The default is to use the firmware that is marked current in the firmware database (see the
showfirmwaredb command).
–minlevel <revlevel>
Minimum firmware level that the cage firmware must be at for the upgrade to be allowed. This
option is deprecated and will be removed in a future release.
–maxlevel <revlevel>
Maximum firmware level that the cage firmware must be at for the upgrade to be allowed. This
option is deprecated and will be removed in a future release.


SPECIFIERS
<cagename>
Specifies the name displayed in the Name column using the showcage command.
768 Upgrade Commands


EXAMPLES
The following example displays a cage with firmware level of 4.41 upgrading to 4.42:
cli% upgradecage –a
Upgrading cage cage0 cpuA from rev 1.46 to revision in file
/opt/tpd/fw/cage/dc2/lbod_fw.bin-2.05.
Upgrading cage cage1 cpuA from rev 1.46 to revision in file
/opt/tpd/fw/cage/dc2/lbod_fw.bin-2.05.
Upgrading cage cage2 cpuA from rev 1.46 to revision in file
/opt/tpd/fw/cage/dc2/lbod_fw.bin-2.05.
Upgrading cage cage3 cpuA from rev 03 to revision in file
/opt/tpd/fw/cage/dc3/dc3_fw.bin-04.
Skipping cage cage4 cpuA already up to date at rev 2.05
Skipping cage cage5 cpuA & cpuB already up to date at rev 04


NOTES
• Access to all domains is required to run this command.
• Before executing the upgradecage command, issue the showcage command to obtain the
names of the drive cages in the system.
• When the upgradecage command is issued, the drive cage becomes temporarily unavailable.
The cage automatically restarts following the firmware update, and may briefly disrupt the
loops during the restart. It is recommended showcage be used approximately one minute
after the update completes to ensure both loops to the cage are available again.

 

 

9 thoughts on “HP 3PAR Cage Degraded and the Upgradecage Command”

  1. Hi Rajesh/Alex

    We are facing this problem when one of our cages is showing degraded, with the firmware state of unknown. Support seem unsure that it won’t cause an outage and the notes of the upgradecage command show ” When the upgradecage command is issued, the drive cage becomes temporarily unavailable.” . Can i please ask both of you if you had any outage when you ran the upgradecage command or did you do it whilst working live?

    Thank you!

    Reply
    • Hi Steve,

      I can only say that we never experienced an outage to a cage when performing this work. I’m wondering if they are meaning that you may see one path to the cage go down while that interface is upgraded. Assuming the system has been cabled correctly this wouldn’t be a problem. Of course I have to say go with support but perhaps ask them to escalate the matter. I’ve had some strange advice or statements in the past that were later changed when a more senior person got involved.

      Definitely be interested in hearing what they say if you do escalate.

      Alex

      Reply
  2. Thanks Alex, i have spoken with a Pro Services contact now who believes it will be fine working online, the wording refers to one path at a time. Of course the proof is in the pudding, so I will let you know how it goes when we do the upgradecage.

    Cheers
    Steve

    Reply
    • Exactly what I thought, yes should be fine. I’ve been through so many upgrades all on ‘mission critical’ systems and we never had an issue so I’m sure you’ll be fine.

      Reply
    • Yeah it isn’t the best. Perhaps they are using it as a catchall in case people have not cabled things properly. Either way the power of the Internet to the rescue :)

      Reply
        • My pleasure, glad I could help – check out the 3PAR simulator if you haven’t already, handy to mess around with though it is limited in what can be done with it. Any other questions or issues feel free to get in touch.

          Reply

Leave a Reply to SteveCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.