Stop the event feedback loop, enforce retention, and make collectors optional
Groundwork for a 1,500-device enquiry on #85, and worth doing regardless: two
of these are bugs that have been quietly costing disk since the features
were written.
The manager was filling its own database
Measured on a five-device fleet: the events table held 568,809 rows over 36
days and was 95% of the entire database.
system,info,account 567,814 rows 99.8%
"user admin logged in from 172.24.1.7 via api" 140,068
"user admin logged out from 172.24.1.7 via api" 140,046
A poll opens an API session, RouterOS logs the login and the logout, and the
next logs poll collects those two lines back and stores them. A closed loop:
the manager watching itself connect, once per poll, per device, for ever. At
1,500 devices that is ~4.7 million rows and 3.8 GB a day of pure self-noise.
Now filtered, but narrowly. A human over Winbox, SSH or the web is a
security-relevant event and is kept; so is an API login by any other
account. Only sessions matching the device's own configured API username
over the API transport are dropped, which is exactly the set we create.
Verified on hardware: four collections, 994 lines skipped each, zero new
account events stored.
retention_events_days did nothing
The setting was seeded by the migration and read by nothing, on the largest
table in the database, while retention_clients_days beside it was enforced.
A retention control that does not retain is worse than none, because it is
believed.
Now pruned hourly, in bounded batches so a first run on a long-unpruned
install does not hold locks for the duration. First run here deleted 109,649
rows.
Collectors can be switched off
Asked for by an operator with ~1,500 devices: "the capability to disable
modules like polling clients or neighbor discovery would be helpful to limit
the amount of traffic to and from router."
He is pointing at the right lever. Concurrency changes how much can happen
at once; this changes how much there is to do. Clients, neighbours, logs and
certificates each toggle, defaulting to on so nothing changes for an
existing install.
Measured with clients and logs off:
logs 694 ms -> 1 ms (returns before connecting)
fast 562 ms -> 211 ms (-62%)
At 1,500 devices and the default concurrency that moves the fast queue from
0.43 headroom to 1.14. The slow queue, at 12.9 s a poll, remains the
bottleneck.
Each toggle costs the feature it feeds and the UI says which, because these
are not tuning knobs -- an operator who discovers three weeks later that
topology stopped updating has been misled. Anything unreadable resolves to
enabled: failing towards more polling is recoverable, failing towards less
means believing in data nobody is gathering.
Full preflight green; docs updated with the measurements and an upgrade note
about the one-off delete.
Container image: ghcr.io/2gt-media-group-llc/mikrotik-manager:0.24.19-beta
Documentation: https://2gt-media-group-llc.github.io/mikrotik-manager/