VMware ESXi SD Card Problem with HPE Server

Recently a number of servers in my estate began to flag the following error –

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.

image

All of the servers in question run VMware ESXi 5.5 and are Hewlett Packard Enterprise (HPE) DL360 Gen8 rack mount servers with SD cards as storage for the ESXi image. None of the servers had ceased functioning while running – production services continued and you would be forgiven for thinking it was something you could ignore. The fun begins when somebody reboots one of the servers (perhaps via Update Manager), at that point it will fail to boot.

When connected via the iLO remote console I observed the system to proceed through the initial BIOS loading and then fail to load with an error indicating no operating system could be found. This seemed odd and a re-seat of the SD card did not help which was my first assumption followed by trying a known good SD card with an ESXi image.

After a little reading I gained a better understanding of how the HPE ProLiant server mounts the SD card – it is in fact through the iLO itself. This is where the problem lies – the iLO firmware we were running had an issue and a firmware update was required. When I went into the iLO logs I came across the following –

iLO Event Log Errors

iLO Self Test Results - Error Embedded Flash SD Card

 

These servers weren’t hugely out of date in terms of the iLO firmware (running 2.20 from May 2015) so I tried updating to the latest release we had deployed in production which was 2.30. This should have resolved the issue but unfortunately it did not and I had to dig around till I found a link to version 2.40 (April 2016) of the iLO firmware. Once this firmware was installed the server successfully booted though I did have to remove the power leads, leave it for 5 minutes and then try as the first boot still failed.

Once the server had finished booting into ESXi I checked for any outstanding alerts and was grateful none displayed –

No alerts

2 thoughts on “VMware ESXi SD Card Problem with HPE Server”

  1. I have been trying to track down these symptoms at a couple customer sites. The first one turned out that the SD card was fine, but the configuration was very old, so fortunately I backed up the ESXi config before reboot, then just restored the config afterwards, and then it was fine.
    Looking through the HP advisories, I’m wondering if this article is related…
    https://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=7271241&docLocale=en_US&docId=emr_na-c04996097

    Reply
    • Hi SE,

      Interesting article – it certainly doesn’t relate to my problem but could of course be an issue for others. I’d say the reliance on iLO firmware for SD card boot is a little frustrating at times when a firmware bug causes this sort of issue.

      Alex

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.