Metrics a scraper can actually read, Mod-Gearman, and a much smaller install

0.22.0 rebuilds the metrics side of the agent. /api/v2/openmetrics served the
JSON keys verbatim — names with dots, spaces and colons in them, no types, no
help text, six significant digits — which a strict parser rejected outright.
It now serves a conformant, self-describing OpenMetrics document: every family
named to the grammar, typed, described, carrying its unit, and per-instance
readings collapsed into one family with a core, nic or drive label
instead of one family per core. Every metric name changes, so read the upgrade
note before you upgrade a scraped host.

The second theme is a new transport. GearmanClient makes the agent a
Mod-Gearman worker: a Naemon or Nagios Core keeps scheduling its checks, the
agent pulls them off a gearmand job server and answers them as native
queries, and the monitored host needs no inbound port at all. The same module
submits passive results back into the core's result queue. It ships marked
experimental and has not been exercised at scale: it is verified end to end
against a real job server on both cores, but no one has yet run it across a
large estate, so put it on a slice of your hosts before you move a whole
Mod-Gearman installation onto it.

Then a long tail of security work — the low-severity tier of the audit, in
three parts, plus a second undefined-behaviour sweep and the ten worst
threading bugs — and a packaging release: common code moved out of every
plugin into shared libraries, Windows ARM64 is back, and there is a Raspberry
Pi package.

✨ Highlights

• 📊 A conformant OpenMetrics exposition, and every metric name changes.
  Names follow the grammar (system.cpu.core 0.idle →
  system_cpu_idle_percent{core="0"}), every family carries # TYPE, # HELP
  and a # UNIT where the value is measured in something, counters are typed
  as counters, string readings come back as _info families, values keep their
  full precision and the body ends with # EOF. Per-core, per-NIC, per-drive
  and per-process readings become one family with a label, so sum by (core)
  has something to group on. The JSON endpoints, the dashboard, Graphite,
  collectd and Python submit_metrics report the same keys and values as
  before. openmetrics format = legacy reproduces the old body as a migration
  window. (#1533, #1535, #1538, #1539)
• 📥 New module: GearmanClient, a Mod-Gearman worker and result channel.
  Optional, off by default, and it speaks both flavours — ConSol's Mod-Gearman
  for Naemon and Nagios Enterprises' Nagios-Mod-Gearman for Nagios Core 4.5+.
  mode = agent answers for the host it runs on; mode = proxy runs a whole
  hostgroup's checks from one box. Windows and Linux. Experimental and not
  yet tested at scale — try it on a few hosts first. (#1542)
• 🔒 A settings source, include or attachment that is not https:// is
  refused. The remote store is the agent's whole configuration, re-read at
  boot and on every housekeeping pass, and over plain http:// nothing
  authenticates the server. Opt back in per host with allow plaintext = true
  in boot.ini; every plaintext fetch is then logged as INSECURE. (#1529)
• 🛡️ The audit's low-severity tier lands, in three parts. A request may no
  longer weaken a credentialed target's transport security or re-address
  submit_smtp mail; script arguments, check_docker's endpoint and the
  remote-connection checks are confined; and the web server sends browser
  hardening headers, negotiates TLS 1.3 on Linux and stops minting a session
  token per request. (#1550, #1551, #1552)
• 🔑 check_tcp and check_ssh verify the server certificate by default.
  verify defaulted to none, so a TLS check handshook against any
  certificate at all and reported ok. It now defaults to peer with ca=
  falling back to the agent's own trust bundle, and the checks gained
  certificate identity, SAN and STARTTLS-service keywords. (#1547)
• 🔁 Agent-to-agent checking works, over the REST API, and the raw protobuf
  web API is gone. POST /query.pb let a caller write the header the core
  reads its identity from; its only consumer was NSCPClient, which had never
  actually worked. check_remote_nscp and friends now go through
  /api/v2/queries, authenticated. (#1552)
• 🧪 New modules and check commands are marked experimental. A statement
  about stability, not about breakage: nscp test, the web UI, the REST API
  and the reference docs all say which checks may still change their options,
  keywords or output. (#1543)
• 🎨 The prompt and the web UI paint filter expressions. Both now read the
  check's own vocabulary and colour a filter=, warning= or detail-syntax=
  as you type it — a keyword the check does not offer is red before the check
  is ever run. A new GET /api/v2/queries/{query}/help is what the browser
  reads. (#1540, #1541)
• ⏱️ Reloads wait for the checks that are running. A settings reload holds
  new checks off a module while it applies the new configuration, a module can
  no longer unload or restart itself from inside a request it is serving, and
  connect and TLS handshake now count against a client's configured timeout.
  (#1531)
• 📦 A fifth off the Linux install, and much more off Windows. Code every
  plugin compiled privately moved into shared libraries, and OpenSSL ships as
  DLLs instead of being linked into each of NRPE, NSCA, check_mk, the web
  server and the HTTP clients. (#1544, #1556)
• 🪟 Windows ARM64 is packaged again, and there is a Raspberry Pi package.
  NSCP-<version>-ARM64.msi/.zip are back (without PythonScript, and the
  ARM64 MSI does not bundle the VC runtime), and
  NSCP-<version>-debian-trixie-arm64.deb covers Raspberry Pi OS 64-bit and
  Debian 13 arm64. (#1546)

🔍 Detailed changes

📊 OpenMetrics — named, typed, described and labelled

The endpoint used to paste the JSON keys into the exposition verbatim. Names
carried ., %, spaces and colons (system_mem_commited.avail,
system_cpu_core 0.idle, disk_free_C:.total); there was no # TYPE and no
# EOF; values were truncated to six significant digits, so 16 GB of memory
scraped as 1.6554e+10; monotonic counts were typed as gauges, so rate()
was unsafe on them; and string readings — uptime, boot time, MAC address,
power source — were dropped entirely. A strict parser rejected the body, and
the scenario page told you to repair the names with metric_relabel_configs.

Three things changed, and each of them renames families.

Names follow the grammar. % becomes the word percent, everything else
outside [a-zA-Z0-9_] becomes _, runs collapse, and a name that would not
start with a letter borrows a metric_ prefix.

Every metric carries a description, a type and a unit. # HELP on every
built-in family, # UNIT wherever the value is measured in something — which
renames the family again, since a family that declares a unit has to end in it
(system_mem_physical_total → system_mem_physical_total_bytes). Counters
are typed as counters and carry the reserved _total suffix on their sample;
strings come back as an _info family in the node_uname_info shape.

Per-instance metrics are one family with a label.

# TYPE system_cpu_idle_percent gauge  
system_cpu_idle_percent{core="0"} 93  
system_cpu_idle_percent{core="1"} 91  
system_cpu_idle_percent{core="total"} 95  

The label is core, cpu, nic, zone, battery, exe, disk, drive
or pdh_instance depending on the bundle — pdh_instance rather than
instance, because Prometheus attaches its own instance label to every
sample. Windows and Linux spell a CPU core differently in the JSON key
(core 0 and core_0); neither spelling reaches the label, which is the bare
0 on both, so one query works across a mixed fleet.

Around that:

• /api/v2/metrics?meta=1 serves the same keys and values with their help
  text, unit, type and labels under a metadata object. Without meta the
  endpoint is byte for byte what it was.
• GraphiteClient can send the labels as carbon tags (metric tags = true,
  off by default — a carbon older than 1.1 stores path;core=0 as the name).
• CollectdClient mappings can read the labels and the types. A variable set
  to label:core expands to every value of that label instead of a regular
  expression over flat keys, and auto: sends whatever the producing module
  declared a counter as a DERIVE and everything else as a GAUGE.
• Predefined PDH counters can describe themselves with help and unit
  keys, and Python fetch_metrics accepts a dict per value
  ({"value": 42, "help": "…", "unit": "bytes", "type": "counter"}, plus
  "labels").
• Windows only: system.mem.page.% and system.mem.physical.% report
  different numbers, because they were reporting the wrong thing — both
  divided the commit charge by the commit limit. Alert thresholds tuned
  against the old reading need re-checking.

📥 GearmanClient — Mod-Gearman, either flavour

A Naemon or Nagios Core installation running Mod-Gearman keeps its scheduler,
its check definitions and its escalations; the agent registers for the queues
you name, pulls jobs off gearmand and answers them as native NSClient++
queries. Nothing listens on the monitored host.

The same module submits passive results into the core's result queue
(/settings/gearman/client, channel GEARMAN, command submit_gearman),
which is what lets a Mod-Gearman installation drop NSCA. The two halves are
independent.

Two things to unlearn when writing the check_command on the core, since
neither fails in a way that names the cause: write host=$HOSTADDRESS$, not
-H $HOSTADDRESS$ (a two-character first token puts the argument parser into
key-value mode and the check answers with a help screen), and
warning=load gt 80, not warning=load>80 (> is a metacharacter, and
allow nasty characters is false by default). See
Mod-Gearman for the full
setup on either core.

The module is marked experimental, and that mark is doing real work here:
its settings, queue handling and output may still change, and while it has been
run end to end against a real gearmand on both cores, it has not been tested
at scale — not against hundreds of hosts, a deep job backlog, or a proxy
answering for a large hostgroup. Roll it out to a slice of the estate first and
keep the existing transport until you are satisfied. Reports of how it behaves
on a real workload are exactly what the experimental mark is asking for.
(#1542)

🛡️ Security

The low-severity tier of the audit landed in three parts, alongside a second
undefined-behaviour sweep and the settings/web/build hardening from #1529.

Outbound clients. The request-override guard now covers the keys that
decide how a connection is protected — verify, insecure, no-psk,
ssl, tls-version, ca, certificate, allowed-ciphers, dh — and, for
SMTP, recipient and sender (behind their own allow recipient override).
proxy= and no-proxy= count as moving the request, since a caller-chosen
proxy receives it whole with the configured NRDP token in it.
payload-length is clamped to what each protocol accepts, 65536 for NSCA and
1 MiB for NRPE. (#1526, #1550)

Scripts and check targets. A NUL in a script argument is refused —
CreateProcessW reads the command line as a C string, so it truncated there
and silently dropped every operator-fixed argument after the substitution
point. ext-scr add --import reads only from the script folders. A script now
receives only its own stdin and its own stdout/stderr pipe ends instead of
every inheritable handle of the service. host= on a docker check must match
the configured endpoint. And on Linux, a script section with user,
domain or password set is refused rather than run as the service account:
those keys are implemented by the Windows launcher only, so a script sandboxed
with user = nobody was not sandboxed at all. (#1523, #1551)

The web server. Every response carries X-Frame-Options: DENY,
X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer and a content
security policy, plus Strict-Transport-Security over TLS — so a page that
embeds the agent's UI in a frame will stop working, which is the point.
[/settings/WEB/server] gains tls version (default 1.2+) and
allowed ciphers. Logging out calls DELETE /api/v2/login, so the bearer
token stops working immediately; the in-memory log buffer is capped at 1000
entries, where an unauthenticated peer could previously add one per rejected
request for the life of the process; and nscp web install-ui no longer stages
its download in ${temp}. (#1552)

Elsewhere. The shared /settings/default/password is registered sensitive
by the core rather than by whichever server module happens to be loaded, so
every agent redacts it. Elastic and Op5 submissions log when credentials go
over an unverified https link, as the Icinga, NRDP and NRPE clients already
did — and the Icinga warning, which repeated on every submission, is logged
once per target now. allowed hosts finally understands the * ranges it has
always advertised (192.168.1.*, 10.*, a bare *); such an entry used to
throw out of the address parser and take the whole listener with it. The
Windows release build pins every third-party action to a commit and verifies
every download against .github/dependency-checksums.txt. (#1529, #1532,
#1534, #1537)

Fleet bundle signatures changed shape, and this one needs action. An
Ed25519 signature used to cover the bare SHA-256 digest of a bundle's bytes,
which bound nothing about which bundle those bytes were: an old signed blob
could be re-advertised under a new id, name or version and still verify. It now
covers a canonical descriptor of the bundle's identity, which means the fleet
server must be upgraded before — or at the same time as — the agents. A
mismatch is safe but inert: the agent logs signature verification failed,
keeps the configuration it last applied, and stops picking up changes. (#1536)

🔁 Agent-to-agent checks over REST

POST /query.pb and POST /settings/query.pb are removed. The first handed
the core a message whose header the caller wrote, and the core reads the
calling module and user out of that header. The second had been unreachable for
several releases. Nothing that talks to an agent over HTTP used them: Icinga's
check_nscp_api asks for GET /query/{name}, untouched, and the web UI uses
/api/v2.

Their only consumer was NSClient++ itself, through NSCPClient — and that
never worked: it passed the serialized message as the HTTP request target, so
every request was malformed, and the configured password was never sent.
check_remote_nscp, remote_nscp_query and remote_nscpforward now run the
command through GET /api/v2/queries/{command}/commands/execute. If you have
an NSCP target configured:

exec_remote_nscp and submit_remote_nscp are removed rather than ported:
both posted an NRPE-style string to the protobuf route, which the remote parsed
as an empty message and answered with nothing — and the submit path reported
success regardless. Use NSCA, NRDP or another submit client for passive
results. (#1552)

🧪 Experimental modules and commands

A module or check command that is new enough that its options, filter keywords
and output may still change now says so: nscp test appends (experimental)
in queries, aliases, list and plugins and shows a Status: line in
desc, the web UI shows a chip, the REST API reports an experimental field,
and the reference documentation renders a marker in the command tables and a
note on the command. The recently added check_* commands of CheckDisk,
CheckDocker, CheckNet, CheckNSCP and Scheduler are marked, as are the
CheckMSSQL, CheckMySQL, CheckSecurity, CheckWindowsApps,
GearmanClient and NSCANgClient modules in full. Out-of-tree modules declare
it in module.json. (#1543)

🎨 Filter expressions, painted

The prompt already knew a command name from a typo. It did not know anything
about what came after the =, which is where the mistakes actually are:
filter=fre < 10% is a check that runs, matches nothing and reports OK. The
value of every option that takes a filter expression or a syntax template is
now read as what it is, following the where grammar and the placeholder rules
the engine itself uses — a keyword the check offers is painted, one it does not
is red, before enter is pressed. Filter functions, operators and number+unit
literals each get their own colour.

The web UI's arguments field does the same, from the same registry data:
GET /api/v2/queries/{query}/help returns one check's whole vocabulary — every
option with its default, flags and description, and every filter keyword it
offers. An alias declares no keywords of its own, so the endpoint follows it to
the command it stands for and says which one in keyword_source. (#1540,
#1541)

⏱️ Threading, reloads and timeouts

The ten highest-severity findings of the concurrency audit are fixed, and
several of them are visible from outside:

• A settings reload holds new checks off a module while its loadModuleEx
  applies the new configuration, and waits up to five seconds for checks
  already inside it to return. A reload is no longer instantaneous on a busy
  agent; if a check holds a module longer than that, the reload proceeds anyway
  and logs which module it was.
• Unloading the module that is serving the request is refused instead of taking
  the agent down with it — in practice
  POST /api/v2/modules/WEBServer/commands/unload. Reloading a listener from
  inside a check that the same listener is serving is refused too, instead of
  leaving that listener dead.
• Configuration downloaded over HTTP gives up on a read or write that stalls
  for 30 seconds, instead of waiting indefinitely, and no longer holds the
  settings instance lock while it downloads.
• Submissions over NRPE, NSCA, NSCP and check_mk apply the configured timeout
  to connecting and to the TLS handshake, not just to the exchange.

Underneath: the Lua script manager and the collector pointers are published
atomically so a reload cannot free them under a running check, the scheduler
watchdog no longer re-arms the pool during shutdown, a socket server refuses to
be stopped from one of its own threads, the Python function registry is read
under the GIL, and a CheckEventLog filter object no longer closes a handle it
does not own. (#1531, #1534)

🐛 Bug fixes

• check_nt FILEAGE checks one file instead of a directory of them. It
  mapped onto check_files, which walks a whole tree, so a directory argument
  reported whichever file the walk happened to emit first — not the oldest, not
  the newest, just arbitrary. It maps onto check_single_file now; a directory
  fails with an error.
• The Windows installer no longer rewrites NRPE transport security it did not
  configure. The MSI recognised only insecure = true and
  verify mode = peer-cert, so a listener running TLS without client
  certificates, or one with a hand-written cipher string, fell through and got
  a preset applied over it. (#1562)
• nscp nrpe install writes use ssl, not ssl = true — a key the server
  never reads. On a host where use ssl = false had been set previously, the
  command claimed encryption and client-certificate authentication while the
  listener stayed in plaintext. The legacy cipher default also drops its
  !ADH, which never excluded the anonymous elliptic-curve suites.
• The PDH counter browser narrows on every filter and ignores case.
  --list, --filter and --counter shared one variable, so whichever came
  last won and the others were silently discarded; --filter is repeatable
  now. Matching is case insensitive, so --list disk finds what --list Disk
  finds.
• A round-robin counter with a zero or unparseable buffer size is refused
  and named in the log, instead of being loaded with a buffer that holds
  nothing.
• A collectd value list naming a metric the snapshot does not carry is no
  longer sent as a zero — it reported a measurement nobody took. Such a value
  list is skipped in whole, since a collectd value list is positional.
• check_ping keeps listening for its own echo reply instead of giving up
  on the first reply that arrives, an HTTP status line with a reason phrase
  parses, every line of a remote check result is kept, and a string Python
  cannot encode no longer crashes the agent.
• check_dns's host= description said the wrong thing. (#1545)

📦 Packaging

• Shared runtime libraries. nscp_net.dll (sockets and TLS),
  nscp_client.dll (the sender modules' shared command line), nscp_json.dll
  and OpenSSL as libcrypto-3-x64.dll / libssl-3-x64.dll now ship next to
  nscp.exe, one copy for the whole service, where before NRPE, NSCA,
  check_mk, the web server, the HTTP clients and the checksum checks each
  carried their own. plugin_api.dll absorbed the settings and program-options
  helpers. About a fifth off the install on Debian and RedHat, and more than
  that on Windows. The legacy XP build still links everything statically.
  (#1544, #1556)
• Windows ARM64 is back as NSCP-<version>-ARM64.msi and .zip. Two
  differences from x64: no PythonScript (the package is cross-compiled and
  there is no ARM64 CPython to embed), and the MSI does not bundle the Visual
  C++ runtime, because Microsoft ships no ARM64 merge module for this toolset —
  install
  vc_redist.arm64.exe
  first on a fresh machine. (#1546)
• A Raspberry Pi OS package, NSCP-<version>-debian-trixie-arm64.deb,
  built on Debian 13 for Raspberry Pi 3 and newer and for Debian 13 arm64 in
  general. 64-bit only, and without the managed (C#) plugin API, since Debian
  does not package the .NET SDK. (#1546)
• The web bundle is built with npm ci and gated on npm audit, so the
  bytes in the web zip and in the MSI's web/dist match package-lock.json.
  Pull requests build only the newest RedHat, and a failed dependency download
  fails at the download instead of somewhere later.

⚠️ Upgrade notes

• Every OpenMetrics family name changes. Drop any metric_relabel_configs
  block that rewrote dots to underscores — the agent does that itself now.
  Update dashboards, recording rules and alerts: system_cpu_core 0.idle is
  system_cpu_idle_percent{core="0"}, disk_free_C:.total is
  disk_free_total_bytes{drive="C:"}, workers_jobs is workers_jobs_total.
  Exclude core="total" from anything that aggregates over cores. If
  dashboards cannot be updated first, set openmetrics format = legacy under
  [/settings/WEB/server] as a migration window — it is deprecated and will be
  removed.
• Upgrade the fleet server before or with the agents. Bundle signatures now
  cover the bundle's identity, not only its bytes. A version mismatch is inert
  rather than damaging: the agent keeps the configuration it last applied and
  stops picking up changes.
• A settings source, include or attachment over plain http:// is refused.
  Move it to https://, or set allow plaintext = true under [tls] in
  boot.ini per host.
• check_tcp / check_ssh against an internal or self-signed service now
  fail with tls_handshake_failed. Point ca= at the issuing CA or add
  verify=none. sni= on a non-TLS connection is now rejected rather than
  ignored.
• POST /query.pb is gone. If you have an NSCP client target, remove an
  explicit path or point it at /api/v2/queries, expect TLS unless you set
  no ssl = true, and give the remote's admin user the queries.execute
  grant. exec_remote_nscp and submit_remote_nscp are removed.
• Requests may no longer weaken a credentialed target's transport security
  (verify, insecure, ssl, ca, tls-version, proxy, …) or re-address
  submit_smtp mail. Supply the credentials with the request, configure the
  variant as its own target, or set allow host override = true /
  allow recipient override = true.
• External scripts with user, domain or password set are refused on
  Linux — those keys were never implemented there. Put the identity change in
  the command itself with sudo.
• The web UI cannot be framed any more, and check_nt FILEAGE naming a
  directory now fails instead of reporting an arbitrary file's age.
• A hand-rolled deployment must copy the new shared libraries from the
  installation root alongside modules\*.dll, the OpenSSL DLLs included.
  Installing from the MSI or the Debian/RedHat packages needs nothing.
• On Windows ARM64, install the VC++ ARM64 redistributable before the agent
  on a fresh machine, and use the x64 package under emulation if you need
  PythonScript.

Security notices for this release are on the
security notices page, and the
full list of behaviour changes is on the
upgrading page.

Full Changelog: 0.21.0...0.22.0