petersulyok/smfc v6.3.0

on GitHub

Added

• New NPU fan controller (sixth controller type) that drives one or more IPMI zones from the temperature of Ascend NPUs, e.g. the Atlas 300I Duo, read with npu-smi. A device is an NPU card (npu-smi -i <id>); for a multi-chip card the hottest chip is used, and temp_calc= (minimum/average/maximum) aggregates across cards. The section is [NPU] / [NPU:1] / ... with npu_device_ids= (the card IDs reported by npu-smi info -m, which may not start from 0), npu_smi_path= (a bare command name resolved via PATH or a full path), and npu_smi_timeout= (seconds, guards against a hung npu-smi call). All the shared parameters of the other temperature-driven controllers (ipmi_zone=, temp_calc=, sensitivity=, polling=, steps=, min_temp/max_temp/min_level/max_level or control_function=, smoothing=, error_tolerance=) are supported. The controller shows up in smfc-client and the HTTP exporter like the other fan controllers. A sample configuration is included as config/samples/smfc-sample10.conf: an NPU-only setup with two Ascend cards in IPMI zone 1. Contributed by @akalagov in PR #121.
• H14 motherboards are supported. platform_name=generic_x14 covers both X14 and H14 boards now. Supermicro's 14th generation ships two different BMC firmware types and the board name does not tell you which one you have, so smfc detects it at startup and reports what it found. On boards with the second type, taking the fans over affects every zone on the board rather than only the configured ones - so list all of them in ipmi_zone=, or the ones you leave out stay at whatever level they had. See README chapter 5.
• New smfc_fan_control_held metric on the HTTP exporter, and the matching control_held and control_detail fields in /snapshot. It reports whether smfc is in control of the fans, which means the same thing on every board: FULL fan mode where that is the controlled state, and the per-zone manual latch on X14/H14. smfc_bmc_info carries a new enforces_full_mode label naming which of the two applies. The sample Grafana dashboard shows it as the Fan control panel. smfc-client colours its fan mode line by this state instead of measuring the fan mode against FULL, and prints the reason when control is lost.
• doc/X14H14_MANUAL_FANCONTROL.md: a reference for both firmware types - which one your board runs, how to drive the fans by hand, and the fan and zone layout of the documented boards.
• New [Ipmi] ipmitool_timeout= parameter (seconds, default: 10) that bounds a single ipmitool execution. A wedged /dev/ipmi0 makes ipmitool block forever, which would park the control loop with nothing regulating the fans behind it. A timed-out call is treated as any other IPMI failure: the control loop logs it and retries on the next poll, and the fan mode watchdog re-acquires control. Use 0 to wait indefinitely, which is how smfc behaved before this parameter existed.

Changed

• smfc stops at startup if a configured ipmi_zone= does not exist on an X14/H14 board. It reads the zone count from the board and names the wrong zone in the error, instead of failing on a raw IPMI error code.
• smfc-client reports the actual fan-control state on X14/H14 instead of measuring the fan mode against FULL. It shows manual control when a zone is latched, or the base fan mode when the BMC's own curve is driving the zones, and warns only when a latch is left held while smfc is not running.

Fixed

• Fan control works on X14/H14 motherboards. It did not before: the commands smfc sent were wrong, so the fan level never reached the board and the BMC kept running the fans on its own curve while smfc reported it had taken over. Tested on real hardware with the help of @koenbossers, @daduijker and @johanvdk1 in discussion #106. Alongside making it work, three safeguards now hold on these boards:
  • The fans are handed back to the BMC on every exit path - exit_level=-1 included, and even when the exit level cannot be written - so they are never left frozen with nothing regulating them. Here the exit level is only a transition: the BMC's own curve takes over about a second later, and it does not include disk temperatures (README chapter 1.6).
  • A zone the BMC has forced to 100% after a fan failure is detected: smfc logs which zone is affected, and with enforce_fan_mode=0 it stops with exit code 11 like any other loss of fan control.
  • min_level=0 can no longer stop the fans: while smfc drives them the BMC's thermal protection is suspended, so levels below 5% are raised to 5%.
• An unreachable BMC is not counted as fan mode drift any more. smfc cannot tell whether the fans drifted when the BMC does not answer, so it no longer treats that as observed drift: it does not increment the smfc_fan_mode_enforced_total metric and does not stop the service when enforce_fan_mode=0. It keeps running and retries every loop, logging an error each cycle - so a temporary BMC outage (e.g. a ~90 second BMC reset, during which the BMC itself usually holds the fans at 100%) is ridden out rather than fatal, and control is re-acquired on the first loop after the BMC returns.