Monday, June 16, 2014

ESXi 5.x HP NIC's not found after firmware update.



When applying the HP Smart Update Manager (HP SUM) or the HP Service Pack for ProLiant on your HP servers your Network cards may stop working. In ESXi NIC's will not be found after firmware(in 5.x).

Problem will occur in the models:

    HP NC373T PCIe Multifunction Gig Server Adapter
    HP NC373F PCIe Multifunction Gig Server Adapter
    HP NC373i Multifunction Gigabit Server Adapter
    HP NC374m PCIe Multifunction Adapter
    HP NC373m Multifunction Gigabit Server Adapter
    HP NC324i PCIe Dual Port Gigabit Server Adapter
    HP NC326i PCIe Dual Port Gigabit Server Adapter
    HP NC326m PCI Express Dual Port Gigabit Server Adapter
    HP NC325m PCIe Quad Port Gigabit Server Adapter
    HP NC320i PCIe Gigabit Server Adapter
    HP NC320m PCI Express Gigabit Server Adapter
    HP NC382i DP Multifunction Gigabit Server Adapter
    HP NC382T PCIe DP Multifunction Gigabit Server Adapter
    HP NC382m DP 1GbE Multifunction BL-c Adapter
    HP NC105i PCIe Gigabit Server Adapter

I have seen these problem occur in 3 ways.
  1.     Applying HP Service Pack for ProLiant (HP SPP) Version 2014.02.0
  2.     Applying directly the firmware update CP021404(particularly in DL360-G5)
  3.     Applying HP Service Pack for ProLiant (HP SPP) Version 2014.02.0 (B)
1 - Applying HP Service Pack for ProLiant (HP SPP) Version 2014.02.0

This is HP Advisory

           DESCRIPTION
"On certain HP ProLiant servers, certain HP Broadcom-Based Network adapters listed in the Scope may become non-functional when they are updated with the Comprehensive Configuration Management (CCM) firmware Version 7.8.21 using firmware smart component
If the server has already stopped responding or boots to a black screen, power-cycle the server to recover and then use the smart components CP021847.scexe and CP021848.scexe Version 2.11.20 to remove the CCM from the affected adapters as described below. If this condition continues to occur, contact HP. Please reference Document ID c04258318 when speaking with HP Support"

This issue can be fixed using the CP CP021847.scexe and CP021848.scexe in the ESXi.
Download the CP, upload to your ESXi and then run it(sh ./CPxxxxx).

But I notice that sometimes this cannot fix the problem.  If this not fix the issue with your Network cards, go to step 2 for the solution.

2 - Applying directly the firmware update CP021404(particularly in DL360-G5)

We can apply the Interface Cards firmware update directly in the  ESXi console running "sh ./CP021404.exe"
Download the CP, upload to your ESXi and then run it(sh ./CPxxxxx).

Even HP stated this can only happen when applying the SPP, I have notice this issue also when applying trough the console the previous firmware update(CP021404) in DL360-G5 servers.
The NICs boot code, PXE/UEFI, IPMI, UMP, CLP, iSCSI, NCSI, FCoE and CCM code are corrupted and/or the MAC address for the network adapter have change(in a dual card both have the same MAC Address).  To correct this is not so easy.

Before the solution I will like to thank all guys in the HP Forum that provided information and some of the steps to fix this issue.

To fix the issue we need to fix the NIC/LOM and prepare the Non-volatile Memory for New NIC/LOM.

1. Download all the tools that we need to fix the NIC/LOM

- download FreeDOS
- download XDIAG.exe
- download bc08c740.bin
- read all informations in SETUP.TXT

2. Prepare the FreeDOS.iso
- After downloading open the ISO  with a tool like UltraISO
- Add the XDIAG.exe and the bc08c740.bin to the ISO using UltraISO
- Save the ISO with a new name
- Burn it or mount it with iLO

3. Boot from CD 
4. Run xdiag in engineering mode, type: xdiag -b06eng
5. On command prompt type: device 1
6. On command prompt: nvm fill 0 0x600 0
7. On command prompt: nvm upgrade -bc "/bc08c740.bin" (use the full path to the bin file with "")
8. On command prompt: nvm cfg and type default, then type 16=10 wich sets the BAR size to 32 for this NIC (see the guide).
9. Save
10. Then lets change to the second network port(device), on command prompt type: device 2 and repeat steps 6-8, run the command 1=00:00:18:xx:xx:xx (change the last digit for different MAC Address on device 2.)
11. Save
12. Exit
13. After do a full POWER CYCLE (unplug) of the server
      Note:You can boot again FreeDos and run xdiag in eng mode again and confirm that the MAC and RAM size are still changed.

14. DONE

Double check the NIC's MAC Address(you can do this in ESXi with the vSphere Client) if the values are set to the MAC Address you add in the step 10.

And that's it, this should fix your HP network cards. If you need more details and help, please go to the HP Forum(added above) and read all posts, or drop a message here.

3 - But also this can happen in the new SPP update: Advisory

Any ProLiant server using HP Smart Update Manager from the HP Service Pack for ProLiant (SPP) Version 2014.02.0 (B) with any of the following adapters:
HP NC371i Multifunction Gigabit Server Adapter
HP NC373i Multifunction Gigabit Server Adapter
HP NC324i PCIe Dual Port Gigabit Server Adapter
HP NC326i PCIe Dual Port Gigabit Server Adapter
  DESCRIPTION
"When an attempt is made to rewrite the Universal Management Port (UMP) firmware on certain HP Broadcom adapters using HP Smart Update Manager (HP SUM) from the HP Service Pack for ProLiant (SPP) Version 2014.02.0 (B), the message, "Update returned an error" may be displayed:
This occurs because the flash installer is not setting the force update flag when attempting to rewrite the UMP firmware even when the "action" tag in the discovery XML is set to "rewrite." As a result, when a rewrite is attempted using Broadcom's flash tools (without the "force" flag), it returns a non-zero return code which is treated as an error by the installer and will eventually result in the "Update returned an error" message in HP SUM."

In this case we disable the update in the SPP, and run CP023219 directly in the console.
Download the CP, upload to your ESXi and then run it(sh ./CPxxxxx).

Note: Any of the above solutions/information that I describe here are provided without any warranty. This is just some solutions/steps that I have tried and fixed my issues.. Particularly for my DL360-G5 Servers.
If you not comfortable to use any, please contact HP to give you support.

No comments:

Post a Comment