Netdata 2.11.0 → 2.12.2
4 releases · 5 breaking changes · 1 security fix · 0 advisories fixed
1security5breaking2deprecated39feature38fix
Breaking changes
- breaking Removed a scan of the open cache that ran on every dbengine datafile deletion only to feed a debug check compiled out of release builds. The scan walked the whole clean queue under its lock, which can be large on busy Parents () in 2.12.1
- breaking Ceph: removed a stray option selector from the configuration form, and stopped the Redfish TLS settings from rendering a duplicate tab strip () in 2.12.1
- breaking Fixed go.d collection status reporting: V2 collection status and duration charts were restored, Function failures are reported and the finalizer context honored, job-scoped Function metadata is returned, accepted job activation converges asynchronously, shutdown errors are preserved through… in 2.12.0
- breaking Removed the unused registry_generate_curl_urls function, and corrected stale comments and test diagnostics (daemon). () in 2.12.0
- breaking Removed a per-chart label probe from the pulse child charts, relying on the host label-version compare instead; on a parent with 808 children the probe accounted for about 23% of the main pulse thread and left gaps in the netdata.* charts it emits () in 2.11.1
Security changes
- security SNMP traps: SNMPv3 messages are now rejected and counted as usm_failures when they skip authentication for a user configured to require it, or declare a non-USM security model. Rejected messages are not acknowledged and do not register engine IDs () in 2.12.2
Deprecations
- deprecated Deprecation notice in 2.12.0
- deprecated The legacy RPM packaging code (netdata.spec.in) remains present but unused and deprecated; official RPM packages are built through CPack, as they have been since v2.11.0. It is still expected to be removed in a future release. in 2.12.0
Every release in between
fixFixed a race where a child that disconnected from its Parent could stay live in Netdata Cloud. The node state update was scheduled before the child was marked offline, so under CPU contention the offline state was sometimes never sent ()
otherA Parent now waits 125% of a disconnected child's expected reboot time, capped at 30 seconds, before reporting it offline to Netdata Cloud. Integer arithmetic had reduced the margin to 100%. A child with no recorded reboot time no longer schedules a zero-delay timer, which can cause high CPU usage…
fixFixed alert transitions being lost on the way to Netdata Cloud. A message the MQTT layer refused, for example while offline or with a full buffer, was treated as delivered and removed from the queue; it is now retried. The alert snapshot also could announce more chunks than it sent, and is now…
fixFixed the reconnect backoff to Netdata Cloud. Once the backoff reached its cap, the clamp was returned in seconds but slept as milliseconds, so a disconnected Agent retried about once per second instead of waiting up to the intended maximum (currently 20 minutes) ()
fixFixed the Cloud connection handing out MQTT packet id 1 twice when the packet id counter wrapped around. Two messages in flight with the same id lose their acknowledgement and timeout tracking ()
fixFixed the t-rx timestamp in the Agent's HTTP replies to Netdata Cloud, which had been sent as 0 since v2.0.0 instead of the time the request was received ()
fixFixed alert evaluation keeping obsolete charts alive. Every evaluation pass refreshed the chart's last-access time, so an obsolete chart with an alert attached was never old enough to be cleaned up. On Kubernetes nodes with pod churn this kept growing the number of k8s.cgroup charts, and the…
fixFixed several health persistence bugs: a prepared statement leaked on every alert transition on hosts without an ACLK configuration, 64-bit event ids were bound as 32-bit values, and the next alarm event id fell back to 0 instead of the alarm id when there were no detail rows ()
fixFixed metadata cleanup passes that could stall. A dimension, chart or label cleanup pass never completed if the first row of its snapshot was deleted before being read, and an interrupted context cleanup dropped the context from its queue, leaving the rest of its dimensions unchecked. Failed chart…
fixFixed a deadlock when freeing a dictionary whose delete callback destroys a nested dictionary that still has referenced items. Tier backfill could reach it when a chart of a removed host was freed while one of its dimensions was still referenced ()
otherdbengine: corrupted gorilla-compressed pages whose length is not a multiple of the buffer size are now rejected instead of being read past the end of their allocation ()
fixFixed completed points of higher tiers staying invisible to queries for a full extra tier period. They were published only when a collection landed exactly on a tier boundary, which slow or offset collectors can miss every time; overdue points are now published on the next collected sample ()
otherdbengine: gaps of a single second are now cached like longer ones, so repeated queries over them no longer search the journals again, and a cached gap no longer overlaps the first point of the page that follows it ()
fixFixed the dimensions= filter of /api/v1/context and /api/v1/contexts, which ignored every metric whose name equals its id (most of them) and so returned all dimensions, instances and contexts ()
fixFixed instance@node_id patterns in the weights API (metric correlations, anomaly rates), which matched nothing ()
fixFixed label filters that combine a bare label name with a name=value term, so matching charts are kept while negative name=value terms still take precedence ()
fixFixed /api/v1/info reading the analytics strings (build info, release channel, notification methods, exporting connectors) without their lock, so a request that overlapped an analytics refresh could read freed memory ()
fixFixed queries returning empty rows for the rest of their time range when an older tier ran out of data well before the planned switch to a newer tier. The query now continues into the newer tier, and SES/DES no longer extend stale values to the end of a query whose tier switch failed ()
otherFunction and dynamic configuration requests now accept only strict JSON request bodies (RFC 8259). Any other body, such as YAML, form data or plain text, is rejected with HTTP 400. The dashboard, Netdata Cloud and the Agent's own callers already send JSON. A saved dynamic configuration that is not…
othergo.d.plugin: the body of a Function request is now treated as data up to its end marker. A line in the body that looked like a plugin command could replace the running Function call, cancel another one, or stop the plugin from reading further requests. Malformed or oversized requests are now…
otherExternal plugins: a protocol error, such as a single corrupted line of plugin output, permanently disabled the plugin until the Agent restarted. Protocol errors now follow the same policy as error exits: the plugin is restarted after 10 × update_every seconds, and disabled only if it never…
securitySNMP traps: SNMPv3 messages are now rejected and counted as usm_failures when they skip authentication for a user configured to require it, or declare a non-USM security model. Rejected messages are not acknowledged and do not register engine IDs ()
otherSNMP: community strings and SNMPv3 passphrases are no longer included in the connection details written to debug logs ()
fixDisk statistics: fixed detection of the discard and flush fields in /proc/diskstats. Kernels before 4.18 got six disk_ext.* charts of zeros, and kernels 4.18 to 5.4 got flushes dimensions stuck at zero. The fields are now detected per disk, and flushes dimensions are added when a disk starts…
otherproc.plugin: reduced overhead on kernels where some /proc files are not seekable, such as Ubuntu 24.04 HWE (6.14). Rewinding such a file no longer resolves its name again on every read, cutting proc.plugin busy time by about 9% ()
otherKubernetes state: deleting a node, pod, deployment or cronjob no longer removes the charts of live objects whose names start with the same name, such as pods web-10 to web-19 when web-1 is deleted. Those charts did not come back until the object was recreated or the Agent restarted, and their…
otherlog2journal: JSON input with arrays nested deeper than the parser's maximum depth is now rejected with an error instead of overrunning its key stack ()
otherNetdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
fixGitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
otherGitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
otherCommunity Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
otherDiscord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 2000 engineers are already using it!
fixFixed the Windows Agent crashing at startup with an access violation inside msys-2.0.dll. The bundled MSYS2 runtime carried an incomplete backport of an upstream Cygwin change; packages now ship a runtime rebuilt with the missing upstream fixes, the MSYS2 installer is pinned by tag and checksum,…
fixFixed Windows timezone detection reporting garbage, or reading past its buffer, when the Windows timezone had no match in the IANA mapping. The mapping is now parsed by attribute with bounded copies, the user's region is preferred over the world default as intended, and on failure the existing…
otherSynced the eBPF collector with ebpf-co-re v1.7.0.3 and kernel-collector v1.7.0.4, reworking the disk and process probes for better kernel compatibility, including hosts without BTF ()
fixFixed ebpf.plugin aborting with heap corruption on hosts where the number of possible CPUs exceeds the number of online CPUs, such as VMware guests ()
fixFixed a crash when an eBPF object or BTF data could not be loaded on builds with older libbpf versions ()
fixFixed ebpf.plugin running the wrong action for most command-line options, so options such as --unittest now do what they say ()
breakingRemoved a scan of the open cache that ran on every dbengine datafile deletion only to feed a debug check compiled out of release builds. The scan walked the whole clean queue under its lock, which can be large on busy Parents ()
otherDocker: reduced the CPU load the collector causes in dockerd and containerd, which was significant on Docker Engine 29+ with the containerd image store. The image list is now refreshed at most once every 5 minutes, and the module's default update_every is raised from 1 to 10 seconds ()
fixDocker: fixed image usage metrics on daemons that negotiate an API older than 1.51, where every image was counted as active and dangling images were always reported as 0. Usage is now derived from the container list on every collection ()
otherRedfish: the configuration form now shows the username and password only for the authentication methods that use them ()
fixRedfish: fixed the collector permanently skipping resources whose @odata.id differs from the URI they were fetched from, such as the trailing-slash variants and remapped NetworkAdapters and Storage paths returned by HPE iLO. Affected jobs stayed in a partial state with a standing…
breakingCeph: removed a stray option selector from the configuration form, and stopped the Redfish TLS settings from rendering a duplicate tab strip ()
otherSMBIOS Memory: the collector is now built only on Linux, the only platform that exports the SMBIOS tables it reads. On macOS, FreeBSD and Windows its stock job used to start and fail with read SMBIOS entry point: ... no such file or directory ()
otherNetdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
fixGitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
otherGitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
otherCommunity Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
otherDiscord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 2000 engineers are already using it!
featureNew Collectors: S3 Object Storage and SMBIOS Memory
deprecatedDeprecation notice
featureSensor and hardware Functions — on-demand tables of the full sensor set and the hardware inventory, with enriched detail (models, part and serial numbers, firmware revisions).
featureIndependent sensor threshold health — each sensor's own upper and lower thresholds, as reported by the BMC, drive its health state, rather than a single collector-wide rule.
featureAn on-demand BMC log viewer — the controller's own event log, read through the same Function mechanism, without logging in to the BMC's web interface.
featureHyper-V monitoring in windows.plugin: aggregate VM health counts, per-VM memory and performance, and root-partition virtualization metrics.
featureRemote Desktop Services sessions: aggregate active and inactive sessions, for the remote-access path that Windows fleets actually run on.
featureCPU charts added to the Windows plugin, plus an exclude space metrics on paths setting that keeps selected volumes out of the disk-space charts.
featureCluster Shared Volumes and folder-mounted volumes now get disk-space charts at their mount paths, with filesystem, serial number, read-only, drive type and mount path labels; some of these volumes were missing from performance-counter results. Mount-point discovery and space queries run on worker…
featureNetwork connections topology (topology:network-connections) now works on Windows, bringing the network viewer to Windows hosts.
featureClearer OS labels on Windows: the OS name is now reported as "Microsoft Windows", and new host labels (_os_marketing_version, _os_release, _os_edition, _os_build) let Windows nodes be filtered by version, release or edition. _os_version keeps its existing display value for compatibility.
featureFixes to Windows alerts, to claiming on Microsoft platforms, and to the MSI installer's verbose logs, which no longer contain the claim token.
featureFile secrets are read without elevated privileges, and configured credential-file reads drop privileges before touching the file.
featureCommand secret providers run through nd-run, the agent's mediated execution helper, which gained one-shot unprivileged file operations and opt-in environment preservation to support exactly this.
featureSecret references in discovered jobs stay literal by default, even when the reference comes from an operator-authored discovery rule, so a discovered target cannot make the agent dereference a secret on its behalf. A service-discovery pipeline you trust can opt back in with…
featuresum results stay exact when stored points span two result rows or a storage-tier boundary
featureincremental-sum queries keep their baseline when a result bucket holds only its opening sample
featureanomaly-rate contributor counts stay correct when a query groups data in two passes
featurethe absolute option is applied at a storage-tier switch, so a negative value can no longer slip through
featurecounter resets and anomaly flags are attributed to the result row that holds the stored sample, not to neighbouring rows that reuse or interpolate its value
featureexplicit and relative single-point latest windows derive their result timestamps from the requested window instead of from the edge of retention; before=0 still asks for the database end.
featureYour Context — one shared Markdown document per Space, with section templates for an infrastructure overview, service tiers, known behaviours, SLOs and business context, team preferences, upcoming events and architecture notes. Every save keeps a version you can compare and restore, and Netdata AI…
featureAI Memory — single facts Netdata AI learns as you work with it, such as "remember that the 02:00 disk spike on backup-01 is expected". It tells you when it records one, and the tab lets you read, delete or clear them.
featureConversations survive a closed tab. Each turn is saved and processed on the server, independently of the browser connection, so closing the tab or losing the connection no longer loses the answer. Stop now cancels generation on the server, and only successful answers are billed.
featureLong conversations keep going. When a conversation approaches the model's context limit, the oldest turns are summarized and the newest kept verbatim, instead of every further turn failing.
featureVisible reasoning. The model's thinking streams into the conversation as it works, collapsed by default and labelled with how long it took.
featureClearer context. Each message shows the context that was sent with it, a conversation started from a chart highlight keeps that time range, your own messages render as Markdown, and long runs of tool calls collapse behind an expandable "N more tools" control.
featureAdded Redfish monitoring of server hardware through the DMTF Redfish API of a baseboard management controller, with sensor and hardware inventory Functions, independent per-sensor threshold health, an on-demand BMC logs viewer, enriched hardware detail, and an operator-facing integration page; the…
featureAdded S3-compatible object storage monitoring with active write/read/list/delete probe checks and cross-site replication timing, then redesigned for safer and clearer operation (go.d/s3check). ()
featureAdded SMBIOS memory inventory monitoring with persistent memory loss detection against an accepted baseline, identifying each slot by its bank and device locator (go.d/smbios_memory). ()
featureAdded OpenTelemetry trace ingestion, trace-aware columnar storage (span extras, trace row index, bloom filters, span combination and rollups) and the otel-traces query Function, with a Functions search view that breaks each trace's spans down by service, an embedded window aggregate, and an…
featureAdded Hyper-V monitoring (aggregate VM health, per-VM memory and performance, root-partition virtualization metrics) and Remote Desktop Services session monitoring to the Windows plugin (windows.plugin). ()
featureAdded a full SNMP topology diagnostics layer: per-walk timing and preparation diagnostics, profile context and failure details, topology source and processing diagnostics, recurring per-device diagnostics, bounded retained evidence, retained topology acquisition history, and publication…
featureAdded portable topology diagnostic archives, an archive diagnostic tool, offline diagnostic inspection, hermetic topology graph replay, link inspection by replay index, support-bundle integration, and inspection of diagnostics directly from support bundles (go.d/snmp, go.d/snmp_topology). ()
featureAdded a permanent SNMP troubleshooting data workflow, generic topology response notifications, and validation of topology search labels, aggregation memberships and typed set cells in scalar columns (snmp, topology). ()
featureAdded network connections topology support on Windows (network-viewer). ()
featureAdded independently configured static and SNMP-backed virtual nodes, with configured vnode metadata authoritative for the jobs that reference them; an initially configured or discovered enabled job that references a vnode not yet defined, or an SNMP vnode still acquiring its identity, waits and…
breakingFixed go.d collection status reporting: V2 collection status and duration charts were restored, Function failures are reported and the finalizer context honored, job-scoped Function metadata is returned, accepted job activation converges asynchronously, shutdown errors are preserved through…
breakingRemoved the unused registry_generate_curl_urls function, and corrected stale comments and test diagnostics (daemon). ()
deprecatedThe legacy RPM packaging code (netdata.spec.in) remains present but unused and deprecated; official RPM packages are built through CPack, as they have been since v2.11.0. It is still expected to be removed in a future release.
+107 more items in the release notesotherPreserved exact SUM values when a stored point spans result rows or an automatic storage-tier boundary, and preserved incremental-sum baselines so buckets holding only an opening sample no longer produce empty queries ()
otherKept API timestamps independent of each Agent's retention, so the same explicit or relative request returns the same row timestamps across Agents ()
otherAttributed tier-0 RESET and anomaly evidence to the result row that actually contains the source sample, and fixed anomaly-rate contributor counts in two-pass group-by queries ()
otherApplied options=absolute to points fetched from the incoming tier during a plan switch, so a negative value can no longer cross a tier transition unchanged ()
fixFixed a logical operator in the JSON wrapper so unqueried dimensions are excluded as expected unless all dimensions are requested ()
fixFixed a use-after-free and a fatal in merged extent queries by acquiring page-cache references before publishing page details and synchronizing merged extent query teardown under the extent spinlock ()
fixFixed a double free of stale MRG metrics during journal v2 migration, and an empty flush-batch counter leak that could leave shutdown or datafile rotation waiting indefinitely ()
otherPrevented negative and epoch-based tier retention reporting: no 32-bit truncation when formatting durations, unknown or zero start times treated as unavailable, empty journals ignored, and extrapolated retention capped ()
otherAccepted valid page cadences longer than one day instead of treating them as invalid and emitting false repair telemetry ()
otherAvoided UUID map create/free churn and partition write-lock contention during MRG metric lookups, and included full cache and page identity in PGC fatal messages to make cache issues diagnosable ()
otherUpdated the bundled SQLite to 3.53.4 ()
fixRepaired the -W createdataset and dbengine stress-test workloads, which crashed on a NULL labels allocator, stamped every generated sample with the wall clock instead of the requested time range, and double-freed state at host teardown ()
featureAdded TLS hostname and IP verification for ACLK HTTPS requests, while preserving insecure-mode behavior ()
otherStopped reporting waived self-signed certificates as certificate verification failures in insecure mode, so later I/O or HTTP errors are reported for what they are ()
otherSwitched the HTTPS and MQTT clients to monotonic, microsecond-based timeouts with an I/O progress watchdog, and bounded pre-CONNACK polling to one second so a quiet peer can no longer delay shutdown by up to 60 seconds ()
otherStopped counting missing MQTT PUBACKs as successful acknowledgements, tracking them with a dedicated timeout counter that is now preserved across disconnects without touching a freed client ()
otherPreserved pooled response buffers and their metadata for compressed ACLK responses, and fixed WebSocket masking of payloads spanning ring-buffer boundaries ()
fixFixed the Windows 10min_cpu_usage alert attaching to service.*_cpu_utilization charts and firing false CRITICAL alarms ()
otherIndexed alerts by name per host, replacing full-host alert scans during health-variable resolution ()
otherAvoided re-initializing alert prototypes when chart metadata has not changed, so continuous child-chart re-registrations no longer trigger repeated health re-evaluation ()
otherBuilt raised-alert summaries only when a notification is actually sent, avoiding per-iteration work and a potential lock-order deadlock with chart removal ()
fixFixed an incorrect error check on SQL statement preparation in the health database code, and removed two unused health_log_detail indexes to reduce parent database storage and B-tree maintenance ()
otherDeferred dictionary destruction while dictionary APIs are in flight, making set/get/delete and traversal safe against concurrent destruction during shutdown ()
fixFixed a use-after-free in dictionary garbage collection when a delete callback re-entered the collector and freed the cached successor mid-walk, and stopped view collections leaving master-deleted items behind ()
otherStopped the obsolete-chart reaper from freeing a chart another subsystem still references, which left an unindexed chart holding its destroy lock and could fatal a collector still writing to it ()
otherDeferred machine-learning worker queue teardown until after all collectors have stopped, so a collector that outlives the shutdown deadline can no longer abort on a destroyed queue mutex ()
fixReset and unblocked SIGPIPE in spawned children, so a child whose pipes are closed exits instead of surviving on EPIPE — which had stranded hundreds of powermetrics processes on macOS — reported failed signal delivery instead of dropping it silently, and made the spawn-server regression tests…
otherProbed for KSM support before enabling page deduplication instead of assuming it, so unsupported kernels no longer report KSM errors, and limited KSM marking to the dbengine page pools that can actually merge ()
otherSkipped sanitizing and re-interning unchanged chart metadata on repeated chart registration, and incremented the label version only on actual label mutations ()
fixFixed a text-buffer cleanup issue and a dyncfg tree sizing race during concurrent configuration updates ()
otherReplaced sprintf() with bounded snprintfz() in inicfg numeric formatting, and replaced dynamic /proc/net/dev path format strings with literal ones ()
otherReduced pulse overhead by running mallinfo2() collection on a configurable, pulse-aligned interval, stamping it after completion so a slow call cannot monopolize the duty cycle, and stopped redundant pulse chart metadata version bumps ()
breakingRemoved a per-chart label probe from the pulse child charts, relying on the host label-version compare instead; on a parent with 808 children the probe accounted for about 23% of the main pulse thread and left gaps in the netdata.* charts it emits ()
otherSilenced a Coverity finding and corrected stale comments and unit-test diagnostics, including documenting health-initialization flag behavior while a child is disconnected ()
fixFixed /proc/interrupts parsing when the first counter is glued to the interrupt ID, which could push changing counter values into dimension IDs and cause unbounded dimension growth ()
fixFixed the charts.d.plugin disabled-chart message pointing to charts.d.plugin.conf instead of the actual charts.d.conf ()
featureSQL Server: added a plan-cache fallback (sys.dm_exec_query_stats) for top-queries when Query Store is unavailable or disabled, and fell back to the built-in system_health session for error-info when no dedicated Extended Events session is configured ()
fixSQL Server: fixed the deadlock and error-info queries on SQL Server 2014 and older, detecting Query Store by catalog column instead of product version, and reading the event_file path from the session catalog instead of assuming it matches the session name ()
featureWindows: added exclude space metrics on paths under [plugin:windows:PerflibStorage] to keep unwanted logical volumes (such as temporary backup mounts) out of disk space collection, added a volume serial_number label, and corrected the misspelled driver_type label to drive_type ()
fixGitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
+12 more items in the release notes