Release v1.27.0

The v1.27.0 release of the Netdata Agent brings dramatic improvements to long-term metrics storage via the database engine, and new dashboard features like a time & date picker for visualizing precise timeframes. Two new collectors bring incredible new value to existing features, including a bit of machine learning magic.

This release contains 8 new collectors, 1 new notification method (2 others enhanced), 54 improvements, 41 documentation updates, and 58 bug fixes.

At a glance

The Netdata Agent now uses SQLite to store host, chart, and dimension metadata. This replaces the only metadata log files, which were located inside of the /var/cache/netdata/dbengine folder for both multihost and legacy children nodes streaming to a parent node. With SQLite powering the metadata log, you should notice faster Agent startups, as it no longer needs to replay metadata log files. The Agent no longer puts archived charts into memory on startup, further reducing memory usage. This is just the first of several improvements to the database engine and metadata log, with more coming in future releases.

The database engine now uses a new extent cache that improves query time by 10% under certain workloads and reduces disk I/O by 10%.

The Netdata Agent's local dashboard has received numerous improvements and bugfixes since v1.26. Perhaps most prominent is the new time & date picker, which helps you select precise timeframes when investigating an anomaly or troubleshooting an incident. See the dashboard repository's releases for the full changelog.

We also introduced two new collectors that monitor the Netdata Agent itself in unique ways. First is the anomalies collector, which uses machine learning (ML) to perform unsupervised anomaly detection on a node running the Netdata Agent. This collector trains itself to understand the baseline of specific charts, then charts anomalous data. A new alarms collector visualizes the volume of Netdata alarms triggered over time.

Acknowledgments

• Ali Dinifar, from ZDResearch, for reporting a stack buffer overflow vulnerability in web_client.c, which our team resolved within 24 hours.
• @ernestojpg for adding the number of allocated/stored objects within each storage to the varnish collector.
• @ernestojpg for adding support for MSE (Massive Storage Engine) to the varnish collector.
• @jurgenhaas for adding allocated space metrics to the oracledb collector.
• @autoalan for fixing a spelling mistake in the haproxy collector README.
• @ysamouhos for fixing a spelling mistake in UPDATE.md.
• @voriol for fixing the Ansible deployment guide.
• @scatenag for adding an option to exclude zero memory allocated users to the nvidia_smi collector.
• @fayak for adding per queue charts to the rabbitmq collector.
• @atnartur for fixing Markdown syntax in the custom dashboard documentation.
• @alexmyczko for removing redundant build dependencies from Debian control file.
• @KickerTom for fixing compilation with HTTPS disabled.
• @hexchain for fixing a database endless loop bug when cleaning obsolete charts.
• @wash2 for fixing the libreswan collector parsing.
• @Saruspete for fixing a platform dependent printf format.
• @KickerTom for fixing an eBPF cross compilation error and updating libnetdata headers to be compatible with C++.
• @WBTMagnum for fixing typos in the README.md.
• @Jiab77 for adding support to hide the SSO iframe.
• @martinpal for adding HBA drives support to the hpssa collector.
• @hamedbrd for fixing response and upstream response time histogram charts in the web_log collector.
• @hamedbrd for adding custom time fields feature to the web_log collector.
• @ski2per for adding directories size collection to the filecheck collector.

Improvements

• Add labels for Kubernetes pods and containers. (#10107, @ilyam8)
• Add plugin and module health entities. (#10041, @thiagoftsm)
• Migrate the metadata log to SQLite. (#10139, @stelfrag)
• Add an extent cache to the database engine. (#10293, @mfundul)
• Added new data query option allow_past. (#10112, @stelfrag)

Netdata Cloud

• Add the ability to query child nodes by their GUID. (#10030, @underhood)
• Add child availability messages to the ACLK. (#9918, @underhood)
• Add a metric showing how long a query spent in the queue. (#10016, @underhood)
• Completely hide the SSO iframe. (#10027, @Jiab77)

Collectors

• Add alarms obsoletion and disable alarms collector by default. (#10375, @ilyam8).
• Add calls for tcp_sendmsg, tcp_retransmit_skb, tcp_cleanup_rcv, udp_sendmsg, udp_recvmsg functions charts to the eBPF collector. (#10360, @thiagoftsm)
• Add two more insignificant warnings to suppress in anomalies collector. (#10369, @andrewm4894)
• Add the number of allocated/stored objects within each storage to the varnish collector. (#10329, @thiagoftsm)
• Add a wireless statistics collector. (#10052, @thiagoftsm)
• Add support for MSE (Massive Storage Engine) to the varnish collector. (#10317, @ernestojpg)
• Remove remove crit from unmatched alarms in the web_log collector. (#10280, @ilyam8)
• Add GPU key metrics (nvidia_smi collector) to dashboard_info.js. (#10230, @ilyam8)
• Add allocated space metrics to the oracledb collector. (#10197, @jurgenhaas)
• Restructure the eBPF collector to improve usability. (#10299, @thiagoftsm)
• Add an anomaly detection collector. (#10060, @andrewm4894)
• Add a Netdata alarms collector. (#10042, @andrewm4894)
• Add a configuration option to exclude users with zero memory allocated to the nvidia_smi collector. (#10098, @scatenag)
• Add per queue charts to the rabbitmq collector. (#10064, @fayak)
• Add support for HBA drives to the hpssa collector. (#10093, @martinpal)
• Update the cgroups collector default filtering by adding pod level cgroups. (#10095, @ilyam8)
• Add a Go version of the CouchDB collector (couchdb). (go.d.plugin/#453, @vlvkobal)
• Add collecting HTTP method per URL pattern (url_pattern option) to the web_log collector. (go.d.plugin/#458, @ilyam8)
• Add custom time fields feature to the web_log collector. (go.d.plugin/#467, @hamedbrd)
• Add a Go version of the PowerDNS Authoritative Nameserver collector (powerdns). (go.d.plugin/#501, @ilyam8)
• Add a Go version of the PowerDNS Recursor collector (powerdns_recursor). (go.d.plugin/#495, @ilyam8)
• Add a Go version of the PowerDNS DNSdist collector (dnsdist). (go.d.plugin/#504, @thiagoftsm)
• Add a Dnsmasq DNS Forwarder collector (dnsmasq). (go.d.plugin/#503, @ilyam8)
• Add collecting directories size to the filecheck collector. (go.d.plugin/#487, @ski2per)
• Add old systemd versions support to the systemdunits collector. (go.d.plugin/#502, @ilyam8)
• Add unmatched lines logging to the web_log collector. (go.d.plugin/#514, @ilyam8)

Notifications

• Add API V2 support to the PagerDuty health integration. (#10189, @thiagoftsm)
• Add threads support to the Google Hangouts health integration. (#10160, @thiagoftsm)
• Add a Opsgenie health integration. (#9879, @thiagoftsm)

Exporting

• Add HTTP and HTTPS support to the simple exporting connector. (#9911, @vlvkobal)

Packaging/installation

• Update React dashboard v2.11. (#10383, @jacekkolasa)
• Update go.d.plugin version to v0.26.2. (#10355, @ilyam8)
• Add numerous improvements to our Docker image. (#10338, @Ferroin)
• Use glibtoolize on macOS instead of regular libtoolize. (#10346, @Ferroin)
• Make the update script significantly more robust and user friendly. (#10261, @Ferroin)
• Update go.d.plugin version to v0.26.1. (#10319, @ilyam8)
• Update React dashboard v2.10.1. (#10314, @jacekkolasa)
• Update go.d.plugin version to v0.26.0 (#10284, @ilyam8)
• Update third-party static dependencies and use alpine 3.12. (#10241, @ktsaou)
• Update React dashboard to v2.9.2. (#10239, @jacekkolasa)
• Update eBPF collector to 0.4.9. (#10202, @thiagoftsm)
• Update go.d.plugin version to v0.25.0 (#10215, @ilyam8)
• Update React dashboard to v2.7.5. (#10179, @jacekkolasa)
• Add ability to use system libwebsockets instead of bundled version. (#9984, @underhood)
• Update the version of libJudy that we bundle to 1.0.5-netdata2. (#10158, @Ferroin)
• Update React dashboard to v2.7.4. (#10122, @jacekkolasa)
• Update go.d.plugin version to v0.24.0 (#10109, @ilyam8)
• Remove redundant build dependencies from Debian control file. (#10085, @alexmyczko)

CI/CD

• Switch to using official Docker actions for GHA CI. (#10364, @Ferroin)
• Explicitly set platform for Docker builds. (#10357, @Ferroin)
• Update distros for CI checks and package builds. (#10123, @Ferroin)
• Remove usage of deprecated GHA syntax. (#10154, @Ferroin)
• Split ReviewDog check to only run when relevant. (#10148, @Ferroin)

Documentation

• Add documentation for time & date picker in Agent and Cloud. (#10347, @joelhans)
• Add paragraph in anomalies collector README to ask for feedback. (#10363, @andrewm4894)
• Fix typo in performance guide. (#10386, @OdysLam)
• Update alarms collector README with fixed image.(#10348, @andrewm4894)
• Update macOS instructions with new Homebrew installation command. (#10379, @ktsaou)
• Update macOS instructions with cmake. (#10295, @joelhans)
• Add guide: Monitor any process in real-time with Netdata. (#10338, @joelhans)
• Improve core documentation to align with recent Netdata Cloud releases. (#10318, @joelhans)
• Add info about network usage requirements for the update script. (#10334, @Ferroin)
• Add new collectors to supported collectors list. (#10310, @joelhans)
• Document the Agent reinstallation process. (#10270, @joelhans)
• Add privacy information about ACLK connection. (#10292, @OdysLam)
• Improve python.d plugin PR checklist README section. (#10302, @andrewm4894)
• Fix a spelling error in the HAProxy documentation. (#10300, @autoalan)
• Fix a spelling error in the update documentation. (#10301, @ysamouhos)
• Add guide: How to optimize Netdata's performance. (#10271, @joelhans)
• Fix a syntax error in bug_report.md. (#10269, @OdysLam)
• Add new issue templates. (#10259, @OdysLam)
• Remove Docker example from update docs and add section to claim troubleshooting. (#10103, @joelhans)
• Improve docs to point users to proper configuration information. (#10254, @joelhans)
• Fix Docs GitHub Action with ignore list and update. (#10002, @joelhans)
• Fix broken links in documentation. (#10253, @joelhans)
• Fix a broken link in the Ansible guide. (#10232, @voriol)
• Add guide: Deploy Netdata with Ansible. (#10199, @joelhans)
• Fix a typo in the streaming doc. (#10225, @ilyam8)
• Fix repeated frontmatter in exporting docs. (#10211, @joelhans)
• Update k8s docs with new Helm repo. (#10172, @joelhans)
• Add a warning to exporting docs about an issue with the newest gRPC versions. (#10194, @vlvkobal)
• Add supported notification platforms to docs. (#10170, @joelhans)
• Add notices to FreeBSD/pfSense docs that they are community-supported. (#10171, @joelhans)
• Fix configuration category in the Prometheus remote write doc. (#10145, @OdysLam)
• Fix broken links. (#10115, @joelhans)
• Add documentation for Cloud Overview. (#10082, @joelhans)
• Update supported collectors list with new collectors. (#10102, @joelhans)
• Fix formatting source code blocks in custom dashboard page. (#10050, @atnartur)
• Add more robust documentation around updates. (#10100, @Ferroin)
• Update CONTRIBUTING.md with new meta title. (#10252, @joelhans)
• Update the Code of Conduct and widen scope to community. (#10186, @OdysLam)
• Update contact information in the Code of Conduct. (#10161, @aabatangle)
• Fix typos in the main README. (#10146, @WBTMagnum)
• Rewrite the repository's main README. (#10108, @joelhans)

Bug fixes

• Fix option parsing in kickstart.sh. (#10396, @Ferroin)
• Fix handling of dependencies on Gentoo. (#10382, @Ferroin)
• Fix crash in the eBPF plugin by initializing variables. (#10395, @thiagoftsm)
• Fix sending chart definition on every data collection in alarms collector. (#10378, @ilyam8)
• Fix a lock check. (#10385, @vlvkobal)
• Fix issue with chart metadata sent multiple times over ACLK. (#10381, @stelfrag)
• Fix a buffer overflow when extracting information from a streaming connection. (#10391, @stelfrag)
• Fix hostname configuration in the exporting engine. (#10361, @vlvkobal)
• Fix use of multiarch/qemu-user-static image for Docker builds. (#10352, @Ferroin)
• Fix handling of self-updating in updater script. (#10352, @Ferroin)
• Fix handling of Python dependency for RPM package. (#10345, @Ferroin)
• Fix handling of PowerTools repo on CentOS 8. (#10334, @Ferroin)
• Fix units and data source exporting options. (#10343, @vlvkobal)
• Fix building libwebsockets properly on macOS. (#10333, @Ferroin)
• Fix exporting config. (#10323, @vlvkobal)
• Fix health by disabling used_file_descriptors alarm. (#10328, @ilyam8)
• Fix GPU data filtering in the nvidia_smi collector. (#10312, @ilyam8)
• Fix username resolution in the nvidia_smi collector. (#10268, @ilyam8)
• Fix compilation with HTTPS disabled. (#10279, @KickerTom)
• Fix hostname when syslog is used in syslog health integration. (#10275, @thiagoftsm)
• Fix kernel crash caused by EBPF in Ubuntu 4.18.0-25 by adding it to the reject list. (#10262, @thiagoftsm)
• Fix streaming buffer size. (#10240, @vlvkobal)
• Fix database endless loop when cleaning obsolete charts. (#10236, @hexchain)
• Disable chart obsoletion code for archived chart creation. (#10231, @mfundul)
• Fix Prometheus remote write exporter so that it doesn't stop when data is not available for dimension formatting. (#10217, @vlvkobal)
• Fix memory calculation by moving shared from cached to used dimension. (#10183, @mfundul)
• Fix parsing in the libreswan collector. (#10190, @wash2)
• Fix an infinite loop in the statsd plugin (#10180, @vlvkobal)
• Fix two bugs related to version handling in install and update code. (#10162, @Ferroin)
• Fix builds using particular versions of Clang. (#10155, @Ferroin)
• Disregard host tags configuration pointer. (#10121, @mfundul)
• Fix platform dependent printf format. (#10120, @Saruspete)
• Fix compile error in CentOS 6. (#10110, @stelfrag)
• Fix cross compilation by properly disabling eBPF detection. (#10034, @KickerTom)
• Fix cgroups collector resolving container names in k8s. (#10072, @ilyam8)
• Fix a compilation warning. (#10320, @vlvkobal)
• Fix UUID_STR_LEN undefined on macOS. (#10313, @underhood)
• Fix python.d plugin runtime chart creation. (#10296, @ilyam8)
• Fix race condition in rrdset_first_entry_t() and rrdset_last_entry_t(). (#10276, @mfundul)
• Fix the data endpoint so that the context param is correctly applied to children. (#10290, @stelfrag)
• Fix Coverity errors (CID 364045,364046). (#10282, @stelfrag)
• Fix the elasticsearch_last_collected alarm. (#10226, @ilyam8)
• Fix spelling error in xenstat.plugin. (#10224, @ilyam8)
• Fix chart filtering. (#10218, @vlvkobal)
• Fix Coverity issues. (#10216, @vlvkobal)
• Fix libnetdata headers to be compatible with C++. (#10185, @KickerTom)
• Fix registry responses to remove caching. (#10181, @cakrit)
• Fix eBPF memory management. (#10096, @thiagoftsm)
• Fix overlapping memory issue. (#10097, @mfundul)
• Fix response and upstream response time histogram charts in the web_log collector. (go.d.plugin/#462, @hamedbrd)
• Fix logs timestamps always in UTC issue in the go.d.plugin (go.d.plugin#460, @ilyam8)
• Fix collecting slave status for MariaDB v10.2.0- in the mysql collector (go.d.plugin#465, @ilyam8)
• Fix cumulative_stats configuration option in the unbound collector (go.d.plugin#478, @ilyam8)
• Fix parsing configuration file (respect 'include-toplevel' directive) in unbound collector (go.d.plugin#480, @ilyam8)
• Fix handling charts with type.id >= 200 (netdata limit) in go.d.plugin (go.d.plugin#472, @ilyam8)
• Fix parsing version query response in the mysql collector (go.d.plugin#498, @ilyam8)
• Fix Netsplits chart dimensions algorithm in the the vernemq collector. (go.d.plugin#511, @ilyam8)
• Fix a typo in dashboard_info.js for VerneMQ. (#10223, @ilyam8)