Netdata 2.12.1 → 2.12.3
2 releases · 2 breaking changes · 1 security fix · 0 advisories fixed
1security2breaking0deprecated3feature33fix
Breaking changes
- breaking Removed the [stream] parent using h2o option. The h2o web server was removed in v2.7.0, and since v2.8.0 a child with this option enabled crashed on every connection attempt to its Parent. A stream.conf that still sets it is accepted and the option is ignored () in 2.12.3
- breaking statsd.plugin: with [statsd] set charts as obsolete after enabled (off by default), the Agent no longer crashes the first time an idle metric is removed. Removal also no longer leaks the metric's values, and metrics used by synthetic charts are kept. The private chart count no longer wraps around… in 2.12.3
Security changes
- security SNMP traps: SNMPv3 messages are now rejected and counted as usm_failures when they skip authentication for a user configured to require it, or declare a non-USM security model. Rejected messages are not acknowledged and do not register engine IDs () in 2.12.2
Every release in between
fixFixed the MQTT keepalive of the Cloud connection. It was only checked when the connection was idle, so a link with steady inbound traffic never noticed an unanswered ping and could stay stalled indefinitely. The ping and acknowledgement deadlines also followed the wall clock, so a clock step…
otherThe Agent's HTTPS client for Netdata Cloud now checks the chunk sizes of chunked responses strictly and caps such responses at 1 MiB. A malformed or very large chunk size could make the Agent exit while allocating memory for it ()
fixFixed use-after-free crashes when netdatacli remove-stale-node or the orphan and ephemeral host cleanup removed a host that was still in use, either by a child reconnecting to it or by a plugin that had it selected as a virtual node. A child that reconnects during a removal is now asked to retry,…
breakingRemoved the [stream] parent using h2o option. The h2o web server was removed in v2.7.0, and since v2.8.0 a child with this option enabled crashed on every connection attempt to its Parent. A stream.conf that still sets it is accepted and the option is ignored ()
featureAccess lists: a client whose IP matches a negated entry, such as !203.0.113.7 in allow connections from = !203.0.113.7 *, is now denied without its hostname being checked against the list. This applies to every access list in netdata.conf when DNS checks are enabled, including the default…
otherURLs with malformed percent escapes are now rejected with HTTP 400. A % had to be followed by two hex digits but was not checked, and a URL that failed to decode was served using the part decoded so far, so /api/v1/info%zz was answered as /api/v1/info. A request containing a literal % that is not…
otherThe MCP API key and the management API key are now compared in constant time, so response timing no longer reveals how close a guessed key is ()
otherMCP: JSON-RPC batches are limited to 64 requests and 16 MiB of responses, and WebSocket messages to 1 MiB, the same as HTTP requests. Requests whose id is not a string, number or null are rejected ()
otherX-Forwarded-For: the Agent now keeps only the first address of the header, and only when it is a valid IPv4 or IPv6 address. This is the forwarded client address shown in the access log and in the audit entries of dynamic configuration changes. Other values, such as unknown, are dropped ()
otherRegistry: the netdata_registry_id identity is now read only from the Cookie header, not from anywhere else in the request, such as the query string. This affects only Agents with the registry enabled, which is off by default ()
otherClaiming: a request to /api/v2/claim or /api/v3/claim with a wrong key no longer regenerates the claim challenge, so it cannot interrupt a claim an administrator is making. The challenge is still regenerated at startup and after a claim with a valid key ()
fixFixed SQLite stalls across the Agent. A global lock was held around SQLite calls, so one slow statement blocked every thread using SQLite, not only those using the same database connection ()
fixCorruption of the metadata database that appears while the Agent is running is now marked for recovery on the first failing query, instead of only at a clean shutdown or never after a crash. The SQLite query statistics no longer count a busy retry that later succeeded as a failed query ()
otherParents using the ram or alloc storage modes no longer shift their children's streamed and replicated history after a gap. Every gap moved all older values one interval earlier and overstated retention by one interval ()
otherdbengine now loads only datafiles with the exact name it writes. A copy such as datafile-1-0000000001.ndf.bak was loaded a second time alongside the real file, and a stray datafile-2-*.ndf stopped the Agent at startup. Other names are now reported as unknown files and skipped ()
fixFixed crashes when the cleanup of obsolete charts freed a chart, or its labels, while health variable lookups, Prometheus export, alert and context API requests or queries were still reading it. These include the Agent exiting on an allocation of a huge size ()
otherdbengine: the Agent no longer hangs at startup when the directory of a tier cannot be created or its initialization thread cannot start. A missing tier above tier 0 now leaves the Agent running with the tiers below it, and a missing tier 0 stops startup with an error ()
fixFixed a crash in metric correlations and other weights queries over two or more nodes on Agents without a dbengine tier, such as a Parent using the ram or alloc storage modes. Such Agents now process the nodes one at a time ()
otherThe summary.nodes[] section of query responses now includes the query time of the last node, which was missing. With options=percentage, the anomaly rate of the dimensions view is no longer inflated. The chart_labels arrays of /api/v1/data now have one entry per dimension, with null where a chart…
other[ml] dimension anomaly score threshold is now limited to 0.01 to 1.00, with a warning in the log when the configured value is changed. Values up to 5.00 were accepted, but anything above 1.00 silently turned off anomaly detection, because a model's anomaly score cannot exceed 100. Such values now…
otherebpf.plugin and ebpf-go.plugin are installed setuid root. When they run with elevated privileges, they now ignore the NETDATA_*_DIR environment variables and use the plugin, configuration and log directories they were built with. Standard installations are not affected. A [directories] override in…
otherRedfish: manager network ports exposed as standard Port resources are now collected, including their link state and speed. They were rejected, leaving the collection permanently partial ()
otherPAN-OS: panos.ipsec.tunnels now counts the established IPsec tunnels instead of reporting the configured ones, and a tunnel with overlapping SAs during rekeying counts once. The remaining SA lifetime now reports the longest valid lifetime of a tunnel instead of a varying or summed value ()
fixgo.d.plugin: fixed collectors such as Cato Networks failing a collection cycle with a host scope conflict when a host's metadata changed while its identity stayed the same ()
otherCato Networks: the topology Function now connects each site to its devices, keeps every observed WAN tunnel instead of merging parallel ones, and marks tunnels as connected, disconnected or unknown. Sites, devices and tunnels carry more detail for troubleshooting, such as HA roles, loss, jitter,…
otherebpf.plugin: the number of tracked PIDs in ebpf.monitoring_pid no longer keeps growing on hosts with short-lived processes. A process that exited before its /proc entry was read was released without being subtracted ()
breakingstatsd.plugin: with [statsd] set charts as obsolete after enabled (off by default), the Agent no longer crashes the first time an idle metric is removed. Removal also no longer leaks the metric's values, and metrics used by synthetic charts are kept. The private chart count no longer wraps around…
otherVirtual nodes: the inactivity timeout set by a collector, such as the one SNMP devices use, works again. It was discarded when the node was defined, so a virtual node never went offline after its collector stopped reporting it. A virtual node taken over again by its local collector after a…
otherSNMP: a poll in which the device answered no request now fails. Profile statistics built from cached metadata were still reported for the device's virtual node, keeping it active after its measurements had stopped ()
fixlog2journal: fixed several JSON parsing and output issues. Very long numbers with an exponent could overflow the number buffer or crash the process, \u000a and \u0009 escapes dropped the rest of the value, unpaired surrogates produced invalid UTF-8, and a line that failed to parse left its fields…
fixgo.d.plugin: fixed two shutdown races in the job manager. A shutdown or failed restart could wait until its deadline instead of finishing, and a Function request that had already been accepted could be reported as rejected because the plugin was stopping ()
featurego.d.plugin service discovery: discovered names and addresses are now encoded as data in the jobs that discovery rules generate. They were inserted into the YAML as text, so an IPv6 address could make a job fail to parse and a name such as on became a boolean. Stock rules use a new job template…
fixSNMP traps and S3: the state files these collectors keep are now read and written safely. Symlinks and special files are rejected, and a temporary file left by a crash no longer blocks later writes ()
fixFixed stock alert and metadata issues: the Portcheck portcheck_connection_fails alert no longer looks up a dimension that does not exist, the Elasticsearch, OpenSearch and Unbound metadata now match the charts they describe, and the Unbound and ethtool module charts are titled "Module ..." again…
fixCrash reports now include a stack trace for SIGABRT raised outside the Agent's own fatal error handling ()
fixOn macOS and Linux builds without libbacktrace, the Agent again saves its status file when it crashes, so the next start reports the exit reason and signal ()
fixFixed a use-after-free at shutdown on Parents with archived children, when loading their contexts outlasted the shutdown wait ()
fixFixed rate-limited log messages being emitted more often than their limit when several threads logged through the same limiter at once ()
featureWhen a collector tries to add a dimension without an id, the fatal error now says so, instead of blaming the destruction of the chart's dimension index ()
fixGitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
+12 more items in the release notesfixFixed a race where a child that disconnected from its Parent could stay live in Netdata Cloud. The node state update was scheduled before the child was marked offline, so under CPU contention the offline state was sometimes never sent ()
otherA Parent now waits 125% of a disconnected child's expected reboot time, capped at 30 seconds, before reporting it offline to Netdata Cloud. Integer arithmetic had reduced the margin to 100%. A child with no recorded reboot time no longer schedules a zero-delay timer, which can cause high CPU usage…
fixFixed alert transitions being lost on the way to Netdata Cloud. A message the MQTT layer refused, for example while offline or with a full buffer, was treated as delivered and removed from the queue; it is now retried. The alert snapshot also could announce more chunks than it sent, and is now…
fixFixed the reconnect backoff to Netdata Cloud. Once the backoff reached its cap, the clamp was returned in seconds but slept as milliseconds, so a disconnected Agent retried about once per second instead of waiting up to the intended maximum (currently 20 minutes) ()
fixFixed the Cloud connection handing out MQTT packet id 1 twice when the packet id counter wrapped around. Two messages in flight with the same id lose their acknowledgement and timeout tracking ()
fixFixed the t-rx timestamp in the Agent's HTTP replies to Netdata Cloud, which had been sent as 0 since v2.0.0 instead of the time the request was received ()
fixFixed alert evaluation keeping obsolete charts alive. Every evaluation pass refreshed the chart's last-access time, so an obsolete chart with an alert attached was never old enough to be cleaned up. On Kubernetes nodes with pod churn this kept growing the number of k8s.cgroup charts, and the…
fixFixed several health persistence bugs: a prepared statement leaked on every alert transition on hosts without an ACLK configuration, 64-bit event ids were bound as 32-bit values, and the next alarm event id fell back to 0 instead of the alarm id when there were no detail rows ()
fixFixed metadata cleanup passes that could stall. A dimension, chart or label cleanup pass never completed if the first row of its snapshot was deleted before being read, and an interrupted context cleanup dropped the context from its queue, leaving the rest of its dimensions unchecked. Failed chart…
fixFixed a deadlock when freeing a dictionary whose delete callback destroys a nested dictionary that still has referenced items. Tier backfill could reach it when a chart of a removed host was freed while one of its dimensions was still referenced ()
otherdbengine: corrupted gorilla-compressed pages whose length is not a multiple of the buffer size are now rejected instead of being read past the end of their allocation ()
fixFixed completed points of higher tiers staying invisible to queries for a full extra tier period. They were published only when a collection landed exactly on a tier boundary, which slow or offset collectors can miss every time; overdue points are now published on the next collected sample ()
otherdbengine: gaps of a single second are now cached like longer ones, so repeated queries over them no longer search the journals again, and a cached gap no longer overlaps the first point of the page that follows it ()
fixFixed the dimensions= filter of /api/v1/context and /api/v1/contexts, which ignored every metric whose name equals its id (most of them) and so returned all dimensions, instances and contexts ()
fixFixed instance@node_id patterns in the weights API (metric correlations, anomaly rates), which matched nothing ()
fixFixed label filters that combine a bare label name with a name=value term, so matching charts are kept while negative name=value terms still take precedence ()
fixFixed /api/v1/info reading the analytics strings (build info, release channel, notification methods, exporting connectors) without their lock, so a request that overlapped an analytics refresh could read freed memory ()
fixFixed queries returning empty rows for the rest of their time range when an older tier ran out of data well before the planned switch to a newer tier. The query now continues into the newer tier, and SES/DES no longer extend stale values to the end of a query whose tier switch failed ()
otherFunction and dynamic configuration requests now accept only strict JSON request bodies (RFC 8259). Any other body, such as YAML, form data or plain text, is rejected with HTTP 400. The dashboard, Netdata Cloud and the Agent's own callers already send JSON. A saved dynamic configuration that is not…
othergo.d.plugin: the body of a Function request is now treated as data up to its end marker. A line in the body that looked like a plugin command could replace the running Function call, cancel another one, or stop the plugin from reading further requests. Malformed or oversized requests are now…
otherExternal plugins: a protocol error, such as a single corrupted line of plugin output, permanently disabled the plugin until the Agent restarted. Protocol errors now follow the same policy as error exits: the plugin is restarted after 10 × update_every seconds, and disabled only if it never…
securitySNMP traps: SNMPv3 messages are now rejected and counted as usm_failures when they skip authentication for a user configured to require it, or declare a non-USM security model. Rejected messages are not acknowledged and do not register engine IDs ()
otherSNMP: community strings and SNMPv3 passphrases are no longer included in the connection details written to debug logs ()
fixDisk statistics: fixed detection of the discard and flush fields in /proc/diskstats. Kernels before 4.18 got six disk_ext.* charts of zeros, and kernels 4.18 to 5.4 got flushes dimensions stuck at zero. The fields are now detected per disk, and flushes dimensions are added when a disk starts…
otherproc.plugin: reduced overhead on kernels where some /proc files are not seekable, such as Ubuntu 24.04 HWE (6.14). Rewinding such a file no longer resolves its name again on every read, cutting proc.plugin busy time by about 9% ()
otherKubernetes state: deleting a node, pod, deployment or cronjob no longer removes the charts of live objects whose names start with the same name, such as pods web-10 to web-19 when web-1 is deleted. Those charts did not come back until the object was recreated or the Agent restarted, and their…
otherlog2journal: JSON input with arrays nested deeper than the parser's maximum depth is now rejected with an error instead of overrunning its key stack ()
otherNetdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
fixGitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
otherGitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
otherCommunity Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
otherDiscord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 2000 engineers are already using it!