Failed checks that explain themselves, a native Icinga 2 agent, and a Checkmk agent that matches the official one
0.27.0 is about the moment a check fails. The agent now keeps a log of its last failed checks, and can keep the metrics and diagnostics tables (top processes, services in trouble, the last log errors, the usual disk growth spots) from that moment, on the host and for the fleet server. It also adds an experimental IcingaAgent module that joins an Icinga 2 cluster natively, brings the Checkmk agent dump in line with the official agent, and adds pressure metrics, per-volume growth, and Hyper-V over-commitment and switch checks.
✨ Highlights
- 🧾 A log of the last failed checks. The core keeps the last 50 WARNING, CRITICAL and UNKNOWN results from every transport, with output and perfdata. It shows on a new Failed checks page, as a dashboard widget and at
GET /api/v2/failures(#1697). - 🔍 What the system looked like when it failed. With
capture metricsandcapture diagnosticson, each failure keeps the metrics snapshot and small diagnostics tables from that moment. A new Diagnostics page shows them live, and an enrolled agent can send failure context to the fleet server (#1699, #1710). - 🧊
IcingaAgent: NSClient++ as a native Icinga 2 agent (experimental). It speaks the cluster protocol on port 5665, requests its certificate from the Icinga CA and runs the checks the master schedules withcommand_endpoint(#1736). - 📤 The Checkmk agent dump matches the official agent. New Linux
<<<cpu>>>and<<<kernel>>>sections, the full memory data, process owners and exact numbers.nscp mk installsets the agent up from the command line (#1719). - 📊 Pressure metrics and volume growth. Linux pressure stall figures, the Windows processor queue and page faults, the average disk queue, and how much each volume grew in the last hour and day, as metrics and as
check_drivesizekeywords (#1710). - 🖥️ Hyper-V over-commitment and virtual switches.
check_hyperv_hostreports CPU and memory over-commitment, the newcheck_hyperv_switcheschecks every virtual switch, andcheck_hyperv_vmsfinds duplicate VM names (#1718). - 🔒 Security fixes. A Checkmk dump field read from the OS can no longer forge rows or sections (#1735). The Windows web backend moves to Mongoose 7.24 (#1722).
🔍 Detailed changes
🧾 Core — the failed-check log
The log is on by default and lives in memory only. Configure it under [/settings/core/failed checks], or in the new folded Failed-check log settings card on the Failed checks page:
| Key | Meaning |
|---|---|
enabled, max entries, statuses
| Whether to keep the log, how many entries (default 50), which states count. |
record arguments
| none, safe (default: thresholds, filter, syntax and what was checked) or all (every option, with password/token values as ***).
|
capture metrics, capture diagnostics
| Keep the metrics snapshot and the diagnostics tables with each failure. Off by default. |
metrics context budget, max context size
| Bound what the snapshots may hold (default 4 MiB). The oldest are released first and their failures stay in the log marked evicted. |
prune metrics, prune pattern
| Trim each snapshot: zeros, empty, duplicates, or keys containing a text.
|
GET /api/v2/failures takes ?status=, ?command=, ?since= and ?limit=. GET /api/v2/failures/<index>/context serves the snapshot, and DELETE empties the log. The endpoints need the new failures.list / failures.delete grants, which only the built-in full role holds.
🔍 Core — diagnostics
capture diagnostics keeps small tables with each failure, each switched on in the module that produces it:
| Table | Holds |
|---|---|
processes.top_cpu, processes.top_memory
| Name, pid, CPU share and memory. Never a command line. |
services.problems, services.events
| Services not running as they should, and the service manager's crash and restart events. |
eventlog.errors
| The last critical and error events (Application and System logs, or the journal). |
disk.growth_spots
| The size of the temp and log folders, page and swap files, and shadow-copy storage. |
failures.recent
| The last failed checks. Always present. |
GET /api/v2/diagnostics (new diagnostics.list grant) serves the latest snapshot live, and nscp test gains a diagnostics [section] command. An enrolled agent sends at most one failure-context report per failure context interval (default 1h) to a fleet server that asks for one. Set failure context = false under [/settings/fleet] to send none.
🧊 IcingaAgent — a native Icinga 2 agent (experimental)
The module is not loaded by default and does nothing until /settings/Icinga/agent has a parent host and a parent endpoint. It then connects to an Icinga 2 master or satellite, gets a certificate (straight away with a ticket, otherwise after icinga2 ca sign), and runs the checks it is sent. CheckCommands import ifw-api, the same convention Icinga for Windows uses. Checks run as IcingaAgent:<parent endpoint>, so the permission policy decides what the parent may run. tls version and allowed ciphers tune the connection. Install the master's ca.crt, or set parent fingerprint, so the first connection is verified too.
📤 CheckMKServer — official agent sections and nscp mk install
| Section | Change |
|---|---|
<<<cpu>>>, <<<kernel>>>
| New on Linux: load, thread count, /proc/vmstat and /proc/stat.
|
<<<mem>>>
| The whole of /proc/meminfo on Linux, commit limit and virtual memory on Windows, exact values.
|
<<<df>>>
| Built by check_drivesize on Linux, with inode counts (needs CheckDisk).
|
<<<ps>>>
| The user field holds the process owner. Every Windows process has a command. |
<<<services>>>
| Delayed and trigger-start services are auto, as the official agent sends them.
|
<<<check_mk>>>
| OSType, OSName and OSVersion, which become cmk/os_* host labels.
|
nscp mk install --allowed-hosts 10.0.0.5 --local "CPU Load=check_cpu warn=load>80 crit=load>95"
nscp mk show
The install command enables CheckMKServer, CheckSystem and CheckDisk, and adds local checks and MRPE entries. A re-run only changes what it is given.
📊 Metrics — pressure, volume growth and a Metrics page that explains itself
- Pressure.
system.pressure.*adds the Linux pressure stall figures for CPU, memory and I/O, major faults, and the Windows processor queue length and page faults.disk.io.<disk>.avg_queue_lengthjoins the instantaneous queue length on both platforms. - Volume growth.
disk.free.<drive>.used_change_1hand_24hsay how much a volume grew, andcheck_drivesizegains the same keywords:warn=used_change_1h > 2G. They readunknownuntil the agent has a full hour (or day) of history. - Metrics page. The web UI shows each metric's description, labels, kind and unit (
640.5 GiB,34 %), grouped by module and instance (#1713, #1715). check_mssql_countersgainslatch_wait_ms, the average latch wait (#1737).
🖥️ Windows — Hyper-V and SCVMM
check_hyperv_hostgainsrunning_vms,vm_vcpus,cpu_overcommit,memory_assigned,physical_memoryandmemory_overcommit. It now reads the VMs over WMI, so it needs the same rights ascheck_hyperv_vms. When it cannot,guests_knownis false andguests_errorsays why.check_hyperv_switchesis new:type,state,healthandportsfor every virtual switch.check_hyperv_vmsgainsname_count(critical=name_count > 1), andhealthreadsdegraded,minor_failure, ... instead ofhealth_10,health_15, ... (#1718).scripts\check_scvmm_hosts.ps1is a new sample script that checks every host System Center VMM manages (#1720).
🛰️ Fleet and core
- Poll interval bounds.
poll interval,min poll intervalandmax poll intervalunder[/settings/fleet]bound how often the host polls, whatever the server asks for (#1729). - Stored module state is per module.
nsclient.dbused to share one namespace between modules. Each module (or alias) now owns its entries, and existing sessions and bookmarks are carried over on first start (#1726). [/settings/core]keys are known. A misspelled key there now gets a warning naming the key it resembles, andnscp settings --validatefails on it (#1731).- No longer experimental:
check_single_file,check_disk_io,check_disk_health,check_tcp,check_dns,check_http,check_connections,check_ntp_offset,check_nscp_updateand theNSCANgClientmodule (#1714).
🔒 Security
- Checkmk dump injection. A line break in a process command line, a mountpoint or a service name could end a row early and pose as another row or a whole section. Control characters are now replaced with a space (#1735).
- Mongoose 7.24. The Windows web backend no longer silently ignores headers past the 30th. A request with more is now refused (#1722).
- OpenSSL 3.5.10. The twelve November OpenSSL CVEs that scanners flag against the bundled 3.5.9 are not reachable through NSClient++. The notice lists why for each.
🐛 Bug fixes
check_httpandcheck_nsclient_web_onlinehonourtimeout=. Neither applied it, so a server that accepted the connection and never answered held the check forever (#1707).check_patch_age. A host with no install dates reported age-1, which looked freshly patched.ageis nowunknown, with a newage_knownflag (#1734).check_serviceno longer fails every call on a service whose configuration the agent's account cannot read, and the Checkmk<<<services>>>section stays.- Windows Update search. The patch collector could crash when a search was aborted while Windows Update was finishing it. The search now runs to completion on its own thread (#1725).
- Scheduler
reportlists accept spaces and any case:report = critical, warningused to submit only criticals. - Settings. Setting a key to an empty value from a module or script now stores the empty value instead of deleting the key. Lua and Python gain
remove_key(). A fleettimeout = 0and negative sizes are refused. - Module listing. An unloaded
.zipmodule bundle now shows in the full module listing, the REST module list and the web UI (#1727).
📦 Packaging
- check_nsclient 1.2.0 is bundled on Windows and Linux. It adds
nsclient failures, which reads the failed-check log and the context kept with each failure,fleet explain, which asks a fleet server to explain an error, andnrpe, which runs a check over NRPE the waycheck_nrpedoes.
⚠️ Upgrade notes
The full list is on the upgrading page, and the security fixes are on the security notices page.
- Checkmk sites: run service discovery on hosts monitored through NSClient++. The memory service moves to Checkmk's own memory check, new CPU services appear, and process rules that match on the user now see the owner. On Linux, enable
CheckDiskto keep<<<df>>>. A Lua line object set withline:set_line(...)now sends an embedded line break as a space. - Custom roles: the failed-check log needs
failures.list(failures.deleteto clear), and diagnostics needdiagnostics.list. check_hyperv_vmshealth: filters onhealth_0,health_10,health_15orhealth_30must useunknown,degraded,minor_failureornon_recoverable_error.check_patch_age: replaceage < 0orage = -1withage_known = 0.nscp settings --validatenow fails on a misspelled key under[/settings/core]. Fix the spelling it suggests.- Scripts that wrote an empty string to reset a key should call
remove_key()instead. - Windows web server behind a proxy: a request with more than 30 headers is now dropped. Trim what the proxy forwards if requests start failing.
- Downgrading after the first start loses stored module state once: web sessions log out and log bookmarks start over.
Full Changelog: 0.26.0...0.27.0