Please help recover Supermicro X9DRH-7F-JBOD's onboard LSI SAS 2308

Technology Tech Support
1

TL;DR:

I have a Supermicro X9DRD-7LN4F-JBOD motherboard which has an onboard LSI 2308. It won’t show up at boot time unless an add-in LSI SAS card is installed. It locked up doing a firmware update and now is in an unusable state. I’ve tried to erase the flash (-e 6 and -e7, and also via megarec -cleanflash 0), but it fails, as does flashing the firmware or BIOS. When executing sas2flash (or sas2flsh) -listall or -list I get a message that the “Controller is not operational. A firmware download is required.” Providing a proper firmware image to download results in a “Firmware Download Failed!” message and sas2flash exits.

Anyone have an idea how I can recover from this?

Full story:

I have a Supermicro X9DRD-7LN4F-JBOD which has an onboard LSI 2308. It won’t show up at boot time unless an add-in LSI SAS card is installed.

I’ve been using this board for a year or so with two add-in HP H220 cards and the onboard SAS disabled via jumper. I want to free up a PCIe slot and add an HP H222 card. So, I enabled the onboard 2308, and with the two H220s and the H222 card in the system I attempted to update the onboard 2308 to the most recent firmware available from Broadcomm, P20.00.07.00. This is where things went to . When I tried to update the firmware and BIOS, the firmware download failed.

sas2flash.efi -o -b MPTSAS2.ROM -f 9207-4i4e.bin -l   SMX9_SAS_Update.log

   Adapter Selected is a LSI SAS: SAS2308_1(D1)


   Executing Operation: Flash Firmware Image


        Firmware Image has a Valid Checksum.

        Firmware Version 20.00.07.00

        Firmware Image compatible with Controller.


        Valid NVDATA Image found.

        NVDATA Version 14.01.00.00

        Checking for a compatible NVData image...


        NVDATA Device ID and Chip Revision match verified.

        NVDATA Versions Compatible.

        Valid Initialization Image verified.

        Valid BootLoader Image verified.


        Beginning Firmware Download...

        Firmware Download Failed!


   Due to error remaining commands will not be executed.

   Unable to Process Commands.

   Exiting SAS2Flash.

I decided to minimize variables and removed the two H220s and the H222 and flash the onboard 2308 with the same result of “Firmware Download Failed’”

Next I looked at Supermicro’s update script for their P20.00.07.00 distro and saw that they did a -e 7 erase before flashing. Since in my experience there is no need to erase the manufacturing area, and it only results in having to reenter the SAS address (as the onboard 2308 doesn’t have a board assembly or tracer number that need to be reentered), I did an -e 6 erase. This failed as well.

sas2flash.efi -o -e 6 -l smx9_SAS_erase.log 
   Adapter Selected is a LSI SAS: SAS2308_1(D1) 

   Executing Operation: Erase Flash

   Erasing Flash Region...

   ERROR: Erase Flash Operation Failed!

   Resetting Adapter...
   Reset Successful!

   Due to Exception Command not Executed. IOCStatus=0x4, IOCLogInfo=0x0
   Finished Processing Commands Successfully.
   Exiting SAS2Flash.[/CODE]

Next I tried an -e 7 erase, grasping at straws. This operation failed, but with a little bit more info.

sas2flash.efi -o -e 7 -l smx9_SAS_erase7.log 
   Adapter Selected is a LSI SAS: SAS2308_1(D1) 

   Executing Operation: Erase Flash

   Erasing Entire Flash Region (including MPB)...

   ERROR: Erase Flash Operation Failed!

   Resetting Adapter...
   Reset Successful!

   Due to Exception Command not Executed. IOCStatus=0x4, IOCLogInfo=0x0
   Finished Processing Commands Successfully.
   Exiting SAS2Flash.

Now, however, the onboard 2308 seems to be totally hosed. sas2flash -list says a firmware download is required, but the download fails:

sas2flash.efi -list -l hosed.log 
   Adapter Selected is a LSI SAS: SAS2308_1(D1) 

   Controller is not operational. A firmware download is required.
  Enter firmware file name or quit to exit:      Executing Operation: Flash Firmware Image

        Firmware Image has a Valid Checksum. 
        Firmware Version 20.00.07.00
        Firmware Image compatible with Controller. 

        Valid NVDATA Image found. 
        NVDATA Version 14.01.00.00 
        Checking for a compatible NVData image... 

        NVDATA Device ID and Chip Revision match verified.
        Valid Initialization Image verified.
        Valid BootLoader Image verified.

        Chip is in FAULT state. Attempting Host Boot...
        Firmware Host Boot Successful.

        Beginning Firmware Download...
        Firmware Download Failed!

   Due to error remaining commands will not be executed.
   Unable to Process Commands.
   Exiting SAS2Flash.

At this point I figured the onboard flash info was completely hosed, and decided to follow these instructions to try and unhose my board. However, this fails at the “Clear the flash on the card with megarec -cleanflash 0” with:

Chip is in FUSION mode.

Erasing flash chip (16MB)....

Flash chip erasing error

Error in erasing flash chip.
Error code = 524288

Now with another LSI SAS card installed, when it is trying to initialize the onboard 2308 (it apparently gets initialed before the add-in H220), I get the following message:

MPT BIOS fault 11h encountered at adapter PCI(03h,00h,00h)
Firmware Fault Code: 0702h

It then goes on to try and initialize the add-in H220, repeats the same error message, gives option to enter the SAS Configuration Utility. This doesn’t show the onboard 2308 and the status for the H220 is “ERROR.”

As I write this it’s late, and I’m tired and frustrated, and really should be sleeping instead of composing this message, but I’m really hoping I’m not screwed. Anyone have an idea as to how I can get the onboard 2308 functional again?

Thank you very much for taking the time to read this.

2

By now you’re probably aware of the -c option to sas2flash that lets you select which card the commands should apply to, when multiple cards are attached. Otherwise it picks the first card, which depends on the PCIe enumeration sequence. I would still flash with only one HBA in the system at a time. Usually, megarec can rescue things. Were you able to flash the empty SBR? You could also try putting the card into another system to flash.

3

Thank you for your reply.

It’s not a card, it’s built-in to the motherboard.

Yes, I used the ‘-c’ option when I had multiple cards in my system. I was grasping at straws, and removed the other cards to minimize variables that could be responsible for the behavior I was witnessing. I date back to ISA and '386 and resource conflict hell. :yum:

I didn’t venture past the step in @spikebyte’s guide where the flash is erased, ‘megarec -cleanflash 0.’ Especially as the inability to erase the flash with sas2flash was the same problem that I was having in updating the FW/BIOS in the first place. I’m afraid of going further and irreparably hosing the 2308, if it isn’t already.

So, are you saying I should ignore not being able to erase the flash, and proceed to the next steps of @spikebyte’s guide, and flash the recovery ROM, etc?

4

No guarantees, but megarec is certainly able to rescue HBAs when sas2flash isn’t able to. The SBR (serial boot record) is different from the firmware region that’s cleared/flashed by -cleanflash. You may also try using the stock SBR from a 2308-based card. The 2308 on the JBOD board should have come in IT mode stock.

5

I forgot to ask if it’s Ok to use the 2208 recovery ROM when this board was supposed to originally be configured as a 2308. I know that besides cache size the boards are pretty much the same, and the real difference is the added code in the firmware to perform RAID functions, hence we can cross-flash. I also assume the SBR and SPD are the same because as far as I can see, the cross-flash process only wipes the firmware and BIOS of the 2208 before flashing the 2308 firmware, leaving the original SBR and SPD in the IR card alone. Is this correct?

Is there any point in @spikebyte’s instructions where a reboot or power off will render the 2308 irrecoverable?

Lastly, I don’t have an access to a stock LSI-branded 2308-based card from which to obtain the SBR. All of my cards are HP-branded IR cards that I cross-flashed. So I can’t get a stock SBR from my own equipment. Do you know where I can obtain one for a revision D1 chip(PCI3.0)?

@seanho thank you very much for your help! :blush:

6

I was successful in reading the SBR, but reading the SPD failed with:

timeout

SPD Read Failed
Error code = 16384

I, of course, have no idea what the error code means, perhaps the point at which it failed as it is 16K.

I then tried to force it to I2O mode using -i2o, but that resulted in this error:

Not able to unlock the Diagnostic registers.

I put back one of the HP H220 cards that is flashed with IT firmware, and attempted to read the SPD and got the same errors I got when trying to read the SPD from the onboard.

Do you have an SPD from a generic LSI-branded 2308 card? Would it be wise to try and flash it? Would it be wiser to flash the both the SBR and SPD from the same generic card?

Should I just try and flash the recovery ROM and see if that gets me back on track? Since I don’t have functioning firmware as it is, I’d think it wouldn’t hurt, but as you can tell, I’m trying to be careful and not screw up this controller more than it already is.

Thank you for your help, @seanho. Between Reddit, STH, and here, you are the only person who has bothered to try and help me out.

7

So, decided to try and flash the 2208_16.rom recovery ROM, and the flashing failed.

  Chip is in FUSION mode
  Flashing image. Please wait...

Currently flashing componenet = BIOS
Erasing Flash....
Error in programming image into Flash.
Error code = 64
Failed to flash the image. Please retry recovery

Out of curiosity, and wondering if the flash itself is damaged, I tried writing the sbrempy.bin and the original SBR that I saved from my board earlier, and both operations succeeded.

So, I’m totally stuck . :rage: