New database, container and system checks

This release widens what NSClient++ can watch (MySQL and MariaDB, a rebuilt Docker module, six new system checks, and a disk-fill projection that alerts on the trend rather than the threshold).

✨ Highlights

• 🐬 MySQL, MariaDB and Percona monitoring. A new CheckMySQL module adds check_mysql for reachability, version, uptime and connection-pool pressure, and check_mysql_query for thresholding arbitrary SQL — every returned column becomes a filter keyword.
• 🐳 CheckDocker rebuilt, from one command to five. Alongside a much richer check_docker, there is now check_docker_info (daemon health), check_docker_stats (per-container CPU and memory against the limit), check_docker_restarts (crash loops and OOM kills) and check_docker_df (disk usage and reclaimable space). The module now ships in the Windows MSI.
• 🖥️ Six new system checks. check_hostname, check_installed_software and check_kernel_memory land on both platforms, check_hardware on Windows, and check_load and check_kernel_stats bring Windows up to parity with Unix — including synthesised load averages, the saturation signal Windows has never exposed.
• 📈 Disk monitoring that predicts instead of reacts. check_drivesize gains full_in, rate, trend_span and trend_samples, projecting time-to-full from a least-squares fit over a configurable window, so a drive that will fill on Saturday can alert on Wednesday.
• 🌐 Richer network checks. Jitter from check_ping and check_ntp_offset, TLS certificate expiry from check_tcp, an address-family flag across the network checks, ICMP payload size and TTL control, and a parsed SSH identification string.
• 🧵 A thread-safety and plugin-lifecycle pass. Races in plugin dispatch, the scheduler, the web token store and the PDH collector, plus spinning and early-start bugs in the log-file and event-log watchers.

🔍 Detailed changes

🐬 CheckMySQL — new module for MySQL, MariaDB and Percona

Two commands, built against MariaDB Connector/C:

check_mysql keywords: version, version_comment, flavor (mysql, mariadb or percona), uptime (duration-typed, so uptime < 1h catches a restart loop), threads_connected, max_connections and connections_pct. There are no default thresholds — reachable is OK — and a connect failure is UNKNOWN with the driver's message. The USAGE privilege is enough.

check_mysql_query requires query=<SQL> and registers every result column as a filter keyword, the same way check_wmi does, plus a line keyword holding the whole row. A statement returning no result set is UNKNOWN rather than a silent OK.

Connection options on both, with defaults from /settings/mysql:

check_mysql host=db1 port=3306 user=monitor password=secret tls=true
check_mysql_query "query=SELECT COUNT(*) AS n FROM app.jobs WHERE state='stuck'" "critical=n > 0"

host=localhost forces TCP unless socket= is given, and plugin-dir= is there for MySQL 8's caching_sha2_password.

🐳 CheckDocker — modernized, and four new commands

The old module had a single check_docker with eight keywords, a hardcoded /v1.40 API prefix, and a transport that fell back to TCP and tried to DNS-resolve /var/run/docker.sock. It now speaks to the daemon over a real unix socket (or named pipe on Windows), reports transport failures as UNKNOWN with the error instead of an empty WARNING, and exposes five commands:

check_docker gains health, has_health_check, ports, labels and created keywords, resolves IPs across multiple networks, and reports a synthetic missing state for a required container that does not exist. New options are timeout=, all=true (include stopped containers) and a repeatable container=<name> with require-semantics. Only container_state != 'running' is critical by default; it used to be both warning and critical.

check_docker_restarts ships the crash-loop default outright — warning on restart_count > 3 and started < 15m and started >= 0, critical on oom_killed = 1:

check_docker_restarts
check_docker_stats container=api "critical=memory_pct > 90"
check_docker_df "warning=total_reclaimable > 10G"

The endpoint comes from /settings/docker (endpoint, timeout), defaulting to \\.\pipe\docker_engine on Windows and /var/run/docker.sock elsewhere. Podman's compat socket works.

🖥️ CheckSystem — load averages, host identity, hardware and kernel counters

check_load brings Unix-style load averages to Windows. Utilization tells you how busy the CPUs are; load tells you how much work is queued for them, which is the saturation signal utilization alone cannot give — 100% CPU with an empty queue is a busy box, 100% with a deep queue is an overloaded one. There is no Windows equivalent to read, so the 1 Hz collector synthesises it from \System\Processor Queue Length plus cores × CPU-busy, folded into three exponential moving averages with elapsed-time-correct decay. Keywords: load1, load5, load15, load, type, queue, procs_running, procs_total, cores, samples, with percpu=true to divide the averages by core count. When the collector is not running, or has not gathered enough samples yet, the check says so as UNKNOWN rather than reporting zeros.

check_hostname reports the identity drift that silently breaks Kerberos authentication, certificate validation and monitoring host-matching. On Windows it reads GetComputerNameEx and NetGetJoinInformation — no WMI — exposing hostname, dns_hostname, domain, fqdn, join (domain, workgroup, standalone or unknown), join_name, fqdn_consistent and netbios_matches_dns; Unix exposes hostname, fqdn, domain and fqdn_consistent. Comparisons are case-insensitive, the NetBIOS check tolerates 15-character truncation, and a host with no DNS suffix is treated as consistent rather than drifting.

check_hardware answers "is this still the same machine" from WMI: vendor, model, uuid, serial, chassis, chassis_type, chassis_serial, asset_tag, memory, modules, slots, memory_speed and module_list. It is built for pinned expectations rather than thresholds — a changed serial means the box was re-imaged, cloned or replaced, and a drop in modules means a DIMM went missing:

check_hardware "critical=serial != 'CZC1234ABC'"
check_hardware "warning=modules < 4" "critical=memory < 64G"

check_installed_software inventories packages for policy enforcement and for answering "what changed just before this started". On Windows it reads the registry Uninstall hives in both the 64-bit and 32-bit views plus every loaded per-user hive under HKEY_USERS — which catches per-user installs like VS Code and JetBrains IDEs regardless of the service account, and deliberately avoids Win32_Product, whose enumeration triggers an MSI consistency check that can reconfigure every installed package on the host. Keywords are name, version, publisher, install_date, install_date_s, install_location, uninstall_string, size, hive, user, architecture, key, system_component and windows_installer; Unix reads dpkg-query, rpm -qa or pacman -Q and reports manager and status instead of the registry-specific ones. An empty result set is OK by design, which makes it a cheap absence probe, while a broken package database is UNKNOWN so it can never read as a clean pass:

check_installed_software "critical=name like 'Flash'"
check_installed_software "filter=install_date > -7d" "warning=count > 0"

Note that version comparisons are lexical, not semver, and the Windows default filter is system_component = 0 — pass filter=none to include runtime and driver components.

check_kernel_stats and check_kernel_memory cover the failure modes that free-RAM and CPU-percent thresholds miss. check_kernel_stats emits one row per metric from the PDH System set, selected with a repeatable type= (ctxt, syscalls, processes, threads), with name, label, human, rate and current per row; it is the only one of the six that ships default thresholds, warning at 8000 threads and critical at 10000 as a thread-leak guardrail. check_kernel_memory reports pool_paged, pool_nonpaged, cache, page_faults_per_sec, transition_faults_per_sec and hard_faults_per_sec on Windows, and slab, slab_reclaimable, slab_unreclaimable, cache, page_faults_per_sec and major_faults_per_sec on Unix — paged-pool exhaustion from a leaking driver and hard-fault storms are classic server failure modes that a free-memory check reports as healthy. Both sample rate counters over a one-second window.

📈 CheckDisk — time until full

check_drivesize can now alert on where a filesystem is heading:

The estimate is an ordinary-least-squares fit of used bytes over a trend-window (default 24h, per check) projected from current free space — the same idea as Prometheus predict_linear or Zabbix timeleft. A long window measures net growth for capacity planning; a short one turns the same keyword into a burst detector.

check_drivesize drive=C: "warning=full_in < 14d" "critical=full_in < 5d"
check_drivesize drive=/var trend-window=2h "critical=full_in < 4h"

Sampling is a collector configured under /settings/disk with trend interval (default 5m) and trend retention (default 7d), and history survives restarts. A slope is only trusted once there are at least three samples spanning at least three sampling intervals; until then full_in is never, rate is unknown, no perfdata is emitted, and numeric comparisons are false in both directions — use warning=trend_span < 1h to tell "no history yet" from "not growing". Resizing a filesystem discards its history. On a total=true row, full_in is the minimum across matched drives and rate their sum.

⚙️ CheckDisk — configurable collection and one perfdata label per metric (#1392)

Two settings under /settings/disk stopped being hardcoded: collection interval (default 10s) which sets the averaging window for every check_disk_io and check_disk_health rate, and max collection errors (default 10) which bounds how many consecutive failures disable a collection. Previously a single failed initial fetch disabled disk-free collection until the service was restarted.

A perfdata bug hid behind that: a metric registered with neither prefix nor suffix emitted the bare perf-syntax alias, so check_disk_health published both free_pct and percent_disk_time under the label 'C:', and check_disk_io did the same for queue_length and percent_disk_time. Icinga keeps both entries, but anything that keys a series by label — a Graphite metric path, an InfluxDB tag set — collapsed them into one series, so queue depth was recorded as disk-busy percent. Those keywords now carry their own suffixes, and the filter refuses to let two variables claim one label.

check_disk_health's space keywords also stop fabricating data on rows that have no filesystem: they render - with no perfdata rather than a flat 0% series.

🌐 CheckNet — jitter, certificate expiry, address family and ICMP tuning

check_ping's jitter is the mean absolute difference between successive round trips and needs count >= 2; check_ntp_offset's is the RMS variation across sampled offsets, matching what ntpq -p reports, and needs samples=N above the default of 1. size=1472 is the largest payload that fits an untagged 1500-byte IPv4 MTU, which makes it a path-MTU and fragmentation-blackhole test; ttl=1 asserts a directly attached neighbour.

check_tcp now reads the peer certificate straight after the handshake, so ssl_expiry_days works at the default verify=none — reading a certificate is not verifying it. Day counts are floored rather than truncated, so the invariant is that the value is negative if and only if the certificate has expired: 23 hours left reads 0 and expired three hours ago reads -1. It shares its name and perfdata suffix with check_http, so expressions port between them.

address-family accepts aliases (4, v4, inet, 6, v6, inet6, both) and rejects anything else rather than defaulting, because a typo must not silently stop testing the family you asked for. check_dns and check_ntp_offset were hardcoded to IPv4 before this, and check_http now parses bracketed IPv6 literals such as http://[::1]:8080/.

check_ssh splits software from version on the last _ followed by a digit, which keeps OpenSSH_for_Windows_9.5 intact; version always holds the whole field, so it is the safe one to match against. protocol_major < 2 is the SSHv1 test — as strings, 1.99 compares as newer than 2.0.

➕ Optional entries and Unix process detail

ignore-missing=true on check_drivesize, check_single_file and check_files drops paths and drives that do not exist instead of failing, so optional and required entries can share one call. It implies empty-state=ok unless you set that explicitly, since an all-missing check would otherwise trade a false CRITICAL for a false UNKNOWN. Windows require= is deliberately unaffected.

The Unix check_process gains uid, username, ppid, proc_state, elapsed and rss (an alias for working_set, matching the Windows keyword set). proc_state carries the real ps state — running, sleeping, disk_sleep, zombie, stopped, tracing_stop, dead, idle, parked — deliberately separate from the cross-platform state, which cannot express a zombie:

check_process process=app "critical=proc_state = 'zombie'"
check_process process=nfsclient "warning=proc_state = 'disk_sleep'"

Name resolution is opt-in through resolve-owner=true, because getpwuid_r can block for seconds against LDAP or SSSD.

🔒 Security fixes

• Remote settings fetches now verify the certificate. The [tls] defaults in boot.ini shipped as verify mode none with no CA, and the file that transport fetches becomes the agent's entire configuration — including [/settings/external scripts], which is arbitrary command execution by design. The default is now verify mode peer against ${ca-path}. Insecure fetches are still possible but must be configured deliberately.
• The legacy /auth/token endpoint is rate-limited. It called validate_user("admin", ...) directly, bypassing the rate limiter entirely: no per-IP backoff, no credential size cap, and a distinct "403 Invalid password" only for a wrong password, which made it a password-correctness oracle — with the password in the query string, so every guess landed in access logs too. It now delegates to the password-header path, which is rate-limited, size-capped and returns one generic 403 for every failure.
• check_docker's host= is constrained to a local endpoint. It is a check argument, so it comes from anyone who can run the check, and it was passed straight to a transport that calls CreateFileA on Windows. A UNC target such as \\attacker\pipe\x therefore made Windows open an SMB session to an arbitrary host using the service account's credentials — commonly LocalSystem — exposing them for capture or relay.
• Duplicate X-HTTP-Method-Override headers are ignored. The two web backends resolved repeats differently (mongoose took the last copy, Beast the first), so sending both was enough to make a proxy and the backend disagree about the request method — the ACL bypass the full-name match had just closed, reached another way. A single override is still honoured; two or more are ignored.
• The web token and grant stores are mutex-guarded. token_store had no synchronisation despite a comment claiming there was no shared state to race, while token generation, expiry, revocation and grant loading all touch it from request handling. It was safe only by accident of both backends running a single request thread — a session table that starts corrupting the day someone adds a worker pool is not a good failure mode to leave armed.
• Installer TLS. TLS_VERIFY_MODE=peer with real verification, logging and handling for installer settings, and the certificate is exported the same way the client does at startup.

🧵 Stability — threading and plugin lifecycle

• Plugin dispatch no longer mutates the plugin map while reading it. Subscriber resolution used plugins_[id] under a shared lock, and std::map::operator[] default-inserts, so it wrote to the map while every concurrent reader was walking it.
• Unregistering a plugin no longer takes another plugin's commands with it. A later plugin may take over an existing command or alias name; removal matched purely on name, so unloading the original erased the replacement's registration. Aliases are also dropped properly when a plugin unloads, and unregistering an unknown command or scheduler task no longer erases end().
• The CheckLogFile watcher no longer spins on WAIT_FAILED. A folder deleted or renamed between the directory check and FindFirstChangeNotification yields an invalid handle, which makes WaitForMultipleObjects fail for the whole array — including the stop event — and nothing handled that case. The inotify instance is also released on stop, and neither watcher starts a thread without a usable stop primitive.
• The PDH collector's setup lock is acquired and released on one thread. The constructor locked and the worker unlocked, which is undefined behaviour that only appeared to work because boost::shared_mutex does not track ownership.
• The check_mk server and client hold the Lua GIL. Lua shares one lua_State across threads with no internal locking; these two call sites were the ones missing the guard that four others already took.
• Smaller fixes. hlp::buffer copy-assignment no longer leaks the block it overwrites; the storage mutex is taken in load() and save(); the scheduler's handler atomic is initialised explicitly and cleared only after workers are joined; the disk I/O stored-data flag is atomic; the Windows max macro no longer eats an interval bound; commands.hpp is self-contained; fleet content is hashed incrementally instead of by concatenation; tags dropped by the tag repository are reported and logged; and the exec argv is built before forking on POSIX.

🐛 Bug fixes

• check_ping's outbound volume (count × payload) is bounded, so a large size= cannot be turned into a burst generator.
• check_http no longer reports a stale certificate expiry after an HTTPS-to-HTTP redirect.
• check_drivesize's trend is judged by the span of its history, not just the number of samples, and the Unix lookup is exact-mount rather than longest-prefix — which could attribute another filesystem's trend to a drive.
• Float-returning filter functions now evaluate at all, so convert_bytes(value,'MB') > 100 works where it was previously silently false; numeric functions no longer infer string type and order 9 above 100. format_bytes, convert_bytes and scale are registered beyond check_pdh.

📦 Packaging and build

• CheckDocker is added to the Windows MSI and the feature-hint map.
• MariaDB Connector/C builds on Windows against a Windows 10 SDK, rather than picking up the 8.1 SDK that the legacy XP build path installs.
• The constantly-changing generated WiX script moved out of the source tree.

⚠️ Upgrade notes

• Breaking for filter expressions that test the old -1 sentinels. A keyword that genuinely has no value no longer parks -1 in itself; it reports no value, and every numeric comparison against it is false — deliberately including =, !=, in and not in, the way SQL treats NULL.
  A threshold over a missing value simply does not fire, in either direction, so there is no longer a sentinel to point thresholds away from. Presence is tested through the string form instead:

  Command	Keywords	Presence test
  check_ping	jitter, ttl	jitter = 'unknown'
  check_ntp_offset	jitter	jitter = 'unknown'
  check_tcp, check_http	ssl_expiry_days	ssl_expiry_days = 'no certificate'
  check_disk_health	total, free, used, user_free, free_pct, used_pct	free_pct = 'no space data'

  The first three rows replace -1 sentinels, so rewrite any jitter = -1, ttl != -1 or ssl_expiry_days = -1 expression. The check_disk_health row is a different shape of the same fix: on rows with no filesystem behind them (I/O-only and physical-disk rows) those keywords used to report a fabricated 0, so a threshold like free_pct < 10 fired on every one of them and now correctly does not.
  Two upsides come with it. A bare critical=ssl_expiry_days < 30 is now safe on a plain connection — that is what the has_certificate guard existed for, and it stays for back-compat — and an expired certificate still reports its real negative day count instead of being ambiguous with "no certificate at all".
  Sentinels elsewhere are untouched: check_process's uid still reports -1 on synthetic rows, as do the age and headroom keywords in other modules. Only the keywords above changed.

• Perfdata series names change for check_disk_health and check_disk_io. free_pct, queue_length and percent_disk_time now carry their own label suffixes instead of sharing the bare drive label. If you graph through Graphite or InfluxDB these arrive as new series; the old collapsed series held the wrong metric anyway. Icinga users see two correctly named entries where they previously saw two entries under one name.

• Perfdata is omitted for values that were not measured rather than plotted as -1, so those series appear and disappear instead of carrying a fake floor.

• CheckDocker requires Docker API 1.41 or newer (the pinned /v1.40 prefix is gone). Any daemon from Docker 20.10 onwards qualifies.

• check_docker's host= no longer accepts remote or UNC endpoints — only a local named pipe or an absolute socket path. A configuration pointing at a remote daemon will now be refused.

• CheckMySQL is built only where MariaDB Connector/C is available, and its MSI entries ship commented out.

Full Changelog: 0.15.0...0.16.0