CEC and BMC Firmware Operations

Firmware upgrade of BMC and CEC components using BMC can be performed from a remote server using the Redfish interface.

CEC and BMC Firmware Commands

Triggering Secure Firmware Update 

Triggers the secure update and starts tracking the secure update progress.

• Update with HttpPushUri API – Deprecated

  HttpPushUri API is obsolete. NVIDIA recommends users migrate to MultipartHttpPushUri API as support for HttpPushUri API will be discontinued in the future.

  
  

  curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService/update
  

  
  

• Multipart update with MultipartHttpPushUri API:

  curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart -F 'UpdateParameters={};type=application/octet-stream' -F UpdateFile=@<package_path>
  

  
  

Where:

• password – password of root user

• bmc_ip – BMC IP address

• package_path – firmware update package path

Tracking Secure Firmware Update Progress

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks

Find the current task ID in the response and use it for checking the progress:

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<task_id> | jq -r ' .PercentComplete'

Where:

• password – password of root user

• bmc_ip – BMC IP address

• task_id – Task ID

Resetting/Rebooting BMC

curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/Actions/Manager.Reset

Where:

• password – password of root user

• bmc_ip – BMC IP address

Fetching Running BMC Firmware Version

Fetches the running firmware version from the BMC.

• For NVIDIA® BlueField®-3:

  curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware | jq -r ' .Version'
  

  Where:

  • password – password of root user

  • bmc_ip – BMC IP address

• For NVIDIA® BlueField®-2:

  curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory
  

  Fetch the current firmware ID and then perform:

  curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/<firmware_id>_BMC_Firmware | jq -r ' .Version'
  

  Where:

  • password – password of root user

  • bmc_ip – BMC IP address

  • firmware_id – numeric value found in the FwInventory schema only. It is calculated during firmware update by the BMC and used to distinguish between the versions.

Fetching Running CEC Firmware Version

Fetches the running firmware version from CEC.

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/Bluefield_FW_ERoT | jq -r ' .Version'

Where:

• password – password of root user

• bmc_ip – BMC IP address

ForceUpdate

ForceUpdate forces the update procedure to proceed even if the target version is identical.

• When using HttpPushUri API, ForceUpdate is always true (i.e., forces the update procedure to proceed even if the target version is identical)

• When using MultipartHttpPushUri API, ForceUpdate is false (i.e., the update procedure fails if the target version is identical), unless "ForceUpdate":true is included under UpdateParameters:

  curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart -F 'UpdateParameters={"ForceUpdate":true};type=application/octet-stream' -F UpdateFile=@<package_path>
  

  Where:

• • password – password of root user

  • bmc_ip – BMC IP address

Updating BMC

After initiating the BMC secure update, a response similar to the following is received depending on whether HttpPushUri or MultipartHttpPushUri is used:

• HttpPushUri API:

  curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService
  
  {
    "@odata.id": "/redfish/v1/TaskService/Tasks/0",
    "@odata.type": "#Task.v1_4_3.Task",
    "Id": "<id>",
    "TaskState": "Running"
  }
  

  
  

• MultipartHttpPushUri API:

  curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart -F 'UpdateParameters={};type=application/octet-stream' -F UpdateFile=@<package_path> 
  {
    "@odata.id": "/redfish/v1/TaskService/Tasks/0",
    "@odata.type": "#Task.v1_4_3.Task",
    "Id": "<id>",
    "TaskState": "Running",
    "TaskStatus": "OK"
  }
  

  Where: 

  • package_path – the BMC firmware image file including its path

    For example: 

    curl -k -u root:'myP@ssword_12345!' https://10.10.10.10/redfish/v1/UpdateService/update-multipart -F 'UpdateParameters={};type=application/octet-stream' -F UpdateFile=@/root/bf2-bmc-ota-24.01-4-ipn.tar 
    

    
    

The following command is used to track secure firmware update progress:

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<id> | jq -r ' .PercentComplete'

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
100  2123  100  2123    0     0  38600      0 --:--:-- --:--:-- --:--:-- 37910
20

The task has completed when PercentComplete reaches 100.

Since the reboot option is disabled during the update procedure, the following command is used to reboot the BMC:

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<id> | jq -r ' .PercentComplete'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
100  3822  100  3822    0     0  81319      0 --:--:-- --:--:-- --:--:-- 81319
100

curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/Actions/Manager.Reset
{
  "@Message.ExtendedInfo": [
    {
      "@odata.type": "#Message.v1_1_1.Message",
      "Message": "The request completed successfully.",
      "MessageArgs": [],
      "MessageId": "Base.1.13.0.Success",
      "MessageSeverity": "OK",
      "Resolution": "None"
    }
  ]
}

The following commands are used to verify the current BMC firmware version after reboot:

• For BlueField-3:

  curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware | jq -r ' .Version'
  
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
  100   513  100   513    0     0   9679      0 --:--:-- --:--:-- --:--:--  9679
  

  
  

• For BlueField-2:Fetch the firmware ID from FirmwareInventory: curl -k -u root:'<password>' -X GET https:/<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/ { "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory", "@odata.type": "#SoftwareInventoryCollection.SoftwareInventoryCollection", "Members": [ { "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/8c8549f3_BMC_Firmware" … Use the following command with the fetched firmware ID from the previous step: curl -k -u root:'<password>' -X GET https:/<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/8c8549f3_BMC_Firmware | jq -r ' .Version' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 471 100 471 0 0 622 0 --:--:-- --:--:-- --:--:-- 621 bmc-23.04

For BlueField-3 BMC only, when updating to an identical version using MultipartHttpPushUri, it is necessary to include ForceUpdate=true:

curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart -F 'UpdateParameters={"ForceUpdate":true};type=application/octet-stream' -F UpdateFile=@<package_path>

Otherwise, the update procedure fails with the message Component image is identical.

Updating CEC

After initiating the BMC secure update, a response similar to the following is received depending on whether HttpPushUri or MultipartHttpPushUri is used:

• HttpPushUri API:

  curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService
  {
    "@odata.id": "/redfish/v1/TaskService/Tasks/0",
    "@odata.type": "#Task.v1_4_3.Task",
    "Id": "0",
    "TaskState": "Running"
  

  
  

• MultipartHttpPushUri API:

  curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart -F 'UpdateParameters={};type=application/octet-stream' -F UpdateFile=@<package_path>
  {
    "@odata.id": "/redfish/v1/TaskService/Tasks/0",
    "@odata.type": "#Task.v1_4_3.Task",
    "Id": "0",
    "TaskState": "Running",
    "TaskStatus": "OK"
  }
  

  Where:

  • package_path – the BMC firmware image file including its path 

    For example: 

    curl -k -u root:'myP@ssword_12345!' https://10.10.10.10/redfish/v1/UpdateService/update-multipart -F 'UpdateParameters={};type=application/octet-stream' -F UpdateFile=@/root/cec_ota_BMGP-04.0f_debug.bin
    

    
    

The following command is used to track the progress of the CEC firmware update:

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/0 | jq -r ' .PercentComplete'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
100  2123  100  2123    0     0  38600      0 --:--:-- --:--:-- --:--:-- 37910
100

The following command is used to verify the current CEC firmware version after reboot:

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/Bluefield_FW_ERoT | jq -r ' .Version'

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   421  100   421    0     0   1172      0 --:--:-- --:--:-- --:--:--  1172
19-4

Activating New CEC

To activate the new CEC firmware, it is necessary to reset the CEC device. The possible options are explained in the following subsections.

Resetting CEC and BMC Subsystems Using CEC Self-reset Command

Redfish Command

curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT/Actions/Chassis.Reset

Where:

• password – password of root user

• bmc_ip – BMC IP address

Command response options:

• The response in case of success:

  {
    "@Message.ExtendedInfo": [
      {
        "@odata.type": "#Message.v1_1_1.Message",
        "Message": "The request completed successfully.",
        "MessageArgs": [],
        "MessageId": "Base.1.15.0.Success",
        "MessageSeverity": "OK",
        "Resolution": "None"
      }
    ]
  } 
  

  
  

• The response if there is no pending CEC firmware:

  curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT/Actions/Chassis.Reset -d '{"ResetType":"GracefulRestart"}'
  {
    "error": {
      "@Message.ExtendedInfo": [
        {
          "@odata.type": "#Message.v1_1_1.Message",
          "Message": "The requested resource of type ERoT FW named 'Pending-ERoT-FW' was not found.",
          "MessageArgs": [
            "ERoT FW",
            "Pending-ERoT-FW"
          ],
          "MessageId": "Base.1.15.0.ResourceNotFound",
          "MessageSeverity": "Critical",
          "Resolution": "Provide a valid resource identifier and resubmit the request."
        }
      ],
      "code": "Base.1.15.0.ResourceNotFound",
      "message": "The requested resource of type ERoT FW named 'Pending-ERoT-FW' was not found."
    }
  }
  

  
  

• The response if the command is not supported by the current CEC firmware:

  curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT/Actions/Chassis.Reset -d '{"ResetType":"GracefulRestart"}'
  {
    "error": {
      "@Message.ExtendedInfo": [
        {
          "@odata.type": "#Message.v1_1_1.Message",
          "Message": "The action ERoT self-reset is not supported by the resource.",
          "MessageArgs": [
            "ERoT self-reset"
          ],
          "MessageId": "Base.1.15.0.ActionNotSupported",
          "MessageSeverity": "Critical",
          "Resolution": "The action supplied cannot be resubmitted to the implementation.  Perhaps the action was invalid, the wrong resource was the target or the implementation documentation may be of assistance."
        }
      ],
      "code": "Base.1.15.0.ActionNotSupported",
      "message": "The action ERoT self-reset is not supported by the resource."
    }
  }
  

  
  

IPMI Command

ipmitool raw 0x32 0xD2

Command response options:

• If successful, no response is given. Glacier and BMC reset.

• The response if there is no pending CEC firmware:

  Unable to send RAW command (channel=0x0 netfn=0x32 lun=0x0 cmd=0xd2 rsp=0xd6): Cannot execute command, command disabled
  

  
  

• The response if the command is not supported by the current CEC firmware:

  Unable to send RAW command (channel=0x0 netfn=0x32 lun=0x0 cmd=0xd2 rsp=0xd5): Command not supported in present state
  

  
  

Log Event Entries Created per Response Type

• Log event entry created in case of success:

  {
    "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>",
    "@odata.type": "#LogEntry.v1_13_0.LogEntry",
    "AdditionalDataURI": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>/attachment",
    "Created": "<Date>",
    "EntryType": "Event",
    "Id": "<Id>",
    "Message": "The request completed successfully.",
    "MessageArgs": [
      ""
    ],
    "MessageId": "Base.1.0.Success",
    "Name": "System Event Log Entry",
    "Resolution": "Ready for ERoT self Reset",
    "Resolved": false,
    "Severity": "OK"
  }
  

  
  

• Log event entry created if there is no pending CEC firmware:

  curl -k -u root:'<password>' -H "Content-Type: application/json" -X GET https://<bmc_ip>/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/11
  {
    "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>",
    "@odata.type": "#LogEntry.v1_13_0.LogEntry",
    "AdditionalDataURI": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>/attachment",
    "Created": "<Date>",
    "EntryType": "Event",
    "Id": "<Id>",
    "Message": "Awaiting for an action to proceed with installing image '' on 'ERoT'.",
    "MessageArgs": [
      "",
      "ERoT"
    ],
    "MessageId": "Update.1.0.AwaitToUpdate",
    "Name": "System Event Log Entry",
    "Resolution": "Cannot perform ERoT self reset: There is no EC FW pending. Perform an ERoT FW update.",
    "Resolved": false,
    "Severity": "OK"
  }
  

  
  

• Log event entry created if the command is not supported by the current CEC firmware:

  {
    "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>",
    "@odata.type": "#LogEntry.v1_13_0.LogEntry",
    "AdditionalDataURI": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>/attachment",
    "Created": "<Date>",
    "EntryType": "Event",
    "Id": "<Id>",
    "Message": "The action ERoT self Reset is not supported by the resource.",
    "MessageArgs": [
      "ERoT self Reset"
    ],
    "MessageId": "Base.1.0.ActionNotSupported",
    "Name": "System Event Log Entry",
    "Resolution": "Cannot perform ERoT self reset: The action is not supported by the current ERoT version.",
    "Resolved": false,
    "Severity": "OK"
  }
  

  
  

• Possible log event entries created if an error occurs during the BMC shut down procedure (received after a success log event entry):

  {
    "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>",
  ...
    "MessageId": "Base.1.0.InternalError",
    "Name": "System Event Log Entry",
    "Resolution": "ERoT Reset: Failed to close services towards ERoT reset",
    "Resolved": false,
    "Severity": "Critical"
  }
  

  
  

  {
    "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>",
  ...
    "MessageId": "Base.1.0.InternalError",
    "Name": "System Event Log Entry",
    "Resolution": "ERoT Reset: Isolate operation may still be in progress.",
    "Resolved": false,
    "Severity": "Critical"
  }
  

  
  

  {
    "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>",
  ...
    "MessageId": "Base.1.0.InternalError",
    "Name": "System Event Log Entry",
    "Resolution": "ERoT Reset: Failed to unmount file system towards ERoT reset.",
    "Resolved": false,
    "Severity": "Critical"
  }
  

  
  

  {
    "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>",
  ...
    "MessageId": "Base.1.0.InternalError",
    "Name": "System Event Log Entry",
    "Resolution": ERoT Reset: Failed to unmount file system. It is still mounted as read-write (rw),
    "Resolved": false,
    "Severity": "Critical"
  }
  

  
  

Resetting CEC and BMC Subsystems Using IPMI I2C Command over SMBus Channel Connected to PCIe Golden Finger

ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x03 0xFE
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x01 0xFE
sleep <100ms>
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x01 0xFF
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x03 0xFF

ipmitool raw <netfun> <cmd> <bus-id> <addr> <read-count> <write-data1> <write-data2>

Resetting Entire BlueField

This option typically involves a full power cycle of the host platform.

CEC Background Update Status

BMC and CEC have an active and inactive copy of the same firmware image on their respective firmware SPI flash. The firmware update is performed on the inactive copy, and on a successful boot from the newly updated and active image, the inactive image (e.g., the previous active image) is updated with the latest image.

To check the status of the background update:

curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT 
... 
  "Oem": { 
    "Nvidia": { 
      "@odata.type": "#NvidiaChassis.v1_0_0.NvidiaChassis", 
      "AutomaticBackgroundCopyEnabled": true, 
      "BackgroundCopyStatus": "Completed", 
      "InbandUpdatePolicyEnabled": true 
    } 
  } 
… 

Possible Error Codes

Last updated: December 17, 2024