VictoriaMetrics 1.148.8 → 1.153.0
5 releases · 1 breaking change · 10 security fixes · 0 advisories fixed
10security1breaking0deprecated15feature44fix
Breaking changes
- breaking BUGFIX: vmsingle, vmselect in VictoriaMetrics cluster and vmctl: accept scientific notation with sub-second precision (e.g. 1.784144612388E9) for timestamp args such as start and end in /api/v1/query_range and --vm-native-filter-time-start and --vm-native-filter-time-end in vmctl. Previously,… in 1.149.0
Security changes
- security SECURITY: vminsert, vmagent and vmsingle: properly apply memory limits to the zstd encoded blocks ingested via native import. This prevents excessive memory allocation during ingestion when handling maliciously crafted requests. See GHSA-8g4f-32hw-vqf8. in 1.153.0
- security SECURITY: vmauth: restrict redirects in the OIDC Discovery HTTP client to stay within the original request host. See GHSA-xxqh-2hcc-9fp6. in 1.153.0
- security SECURITY: upgrade base docker image (Alpine) from 3.24.1 to 3.24.2. See Alpine 3.24.2 release notes. in 1.153.0
- security BUGFIX: vmselect, vminsert, vmagent and vmsingle: ignore empty extra_label, extra_filters and extra_filters[] query args instead of returning an error. This allows dropping client-provided values for these query args via url_prefix in vmauth as described in the security docs. See. in 1.153.0
- security SECURITY: upgrade Go builder from Go1.26.6 to Go1.27.1. See Go 1.27 release notes. in 1.152.0
- security SECURITY: vmauth: fix authorization bypass in JWT-based routing when match_claims values are used. See GHSA-f99m-22fh-qw96. in 1.152.0
- security SECURITY: vmsingle and vmselect in VictoriaMetrics cluster: properly escape server-side errors returned by /metric-relabel-debug since they could contain parts of user input. This prevents XSS attacks on the /vmui/#/relabeling page. See GHSA-vw10j-rqhg-qff9. in 1.152.0
- security SECURITY: vmsingle, vmagent, vminsert and vmselect in VictoriaMetrics cluster: check HTTP Basic Auth credentials from the -httpAuth. command-line flags for cases that do not verify the corresponding *AuthKey command-line flag on its own. Previously, HTTP Basic Auth could be skipped for request… in 1.151.0
- security SECURITY: upgrade Go builder from Go1.26.5 to Go1.26.6. See the list of issues addressed in Go1.26.6. in 1.150.0
- security SECURITY: vmsingle and vmselect in VictoriaMetrics cluster: restrict /api/v1/admin/tsdb/delete_series, /tags/delSeries endpoints to POST method only to prevent some SSRF-based data deletion attacks. See. in 1.149.0
Every release in between
securitySECURITY: vminsert, vmagent and vmsingle: properly apply memory limits to the zstd encoded blocks ingested via native import. This prevents excessive memory allocation during ingestion when handling maliciously crafted requests. See GHSA-8g4f-32hw-vqf8.
securitySECURITY: vmauth: restrict redirects in the OIDC Discovery HTTP client to stay within the original request host. See GHSA-xxqh-2hcc-9fp6.
securitySECURITY: upgrade base docker image (Alpine) from 3.24.1 to 3.24.2. See Alpine 3.24.2 release notes.
otherFEATURE: vmalert: skip a redundant pending state when an alert can be restored directly to firing after restart. See.
otherFEATURE: vmauth: log informative message when a JWT token has no vm_access claim and default_vm_access_claim is not configured, if -logInvalidAuthTokens is enabled. See JWT claim-based request templating or.
otherFEATURE: dashboards/vmauth: update the dashboard to upstream Grafana version, fix typos and inconsistencies, improve resource usage panels. See
otherFEATURE: vmauth: support printing HTTP headers in the Access Logs. Only headers listed in access_log.headers list will be printed in the logs. This feature should help troubleshooting pipelines that rely on HTTP headers. See.
otherFEATURE: vmui: move auto-refresh controls from the header to the Execute button dropdown and display the selected interval in the button text. See.
otherFEATURE: vmstorage in VictoriaMetrics cluster: support ingesting data directly via Prometheus Remote Write v1 protocol for vmstorage. This feature can be enabled by setting -enableIngestionAPI command-line flag and is disabled by default. See remote write directly to vmstorage and.
featureFEATURE: vmauth: add Single sign-on (SSO) support via OpenID Connect (OIDC). vmauth redirects unauthenticated browser requests to the configured Identity Provider (IdP), verifies the IdP response, and sets a session cookie for subsequent requests. See.
featureFEATURE: vmsingle and vmstorage in VictoriaMetrics cluster: add support for opt-in fadvise(FADV_RANDOM) and madvise(MADV_RANDOM) hints for data part files, which can reduce disk I/O and improve page cache efficiency. To enable, set -fs.disableAdviseRandomRead=false. If you experience issues after…
fixBUGFIX: all VictoriaMetrics components: create the Unix domain socket for -httpListenAddr=unix:/path/to/socket with permissions according to the umask of the process. Previously it was always created with 0600 permissions, so processes running under other users, such as a reverse proxy, couldn't…
fixBUGFIX: stream aggregation: serially push samples to stream aggregation jobs. This helps reduce CPU overhead. 11f488d8ff introduced parallel processing for this stage, which introduced extra CPU overhead while only providing negligible sample lag reduction. See.
fixBUGFIX: vmagent: add missing X-Influxdb-Version HTTP header to influxdb API responses. See.
fixBUGFIX: vmagent: disable metrics metadata sending for MDX remote write destinations. See.
fixBUGFIX: vmagent and vmsingle: properly discover *_sd_config targets. Previously scrape targets were preserved until the process restart if *_sd_config returned an empty targets response. See for details. Thanks to @evkuzin for contribution.
fixBUGFIX: vmagent: properly reload SSL certificate for kafka producer and consumer. See.
fixBUGFIX: vmalert: properly reload alert_relabel_configs from the notifier configuration file. Previously, this config was applied only at startup and was not updated on config reload. See.
securityBUGFIX: vmselect, vminsert, vmagent and vmsingle: ignore empty extra_label, extra_filters and extra_filters[] query args instead of returning an error. This allows dropping client-provided values for these query args via url_prefix in vmauth as described in the security docs. See.
fixBUGFIX: vmauth: respect the -enableTCP6 flag when discovering backend IPs via discover_backend_ips or -discoverBackendIPs. Previously, discovery included IPv6 addresses even when IPv6 support was disabled. See.
fixBUGFIX: vmui: fix synchronization of the custom query step with URL changes on the predefined dashboards page. See.
fixBUGFIX: vmsingle and vmselect in VictoriaMetrics cluster: slightly reduce CPU usage during concurrent queries execution. See.
securitySECURITY: upgrade Go builder from Go1.26.6 to Go1.27.1. See Go 1.27 release notes.
securitySECURITY: vmauth: fix authorization bypass in JWT-based routing when match_claims values are used. See GHSA-f99m-22fh-qw96.
securitySECURITY: vmsingle and vmselect in VictoriaMetrics cluster: properly escape server-side errors returned by /metric-relabel-debug since they could contain parts of user input. This prevents XSS attacks on the /vmui/#/relabeling page. See GHSA-vw10j-rqhg-qff9.
featureFEATURE: vmauth: add the deny_paths option to url_map to reject a subset of paths matched by src_paths or other src_* options with 403 Forbidden (401 Unauthorized for anonymous requests) instead of enumerating every allowed path. See. Thanks to @andriibeee for contribution.
otherFEATURE: alerts-health: split TooManyLogs alert into TooManyErrorLogs and TooManyLogs. TooManyLogs now covers only warn-level logs and aggregates them per job and instance. TooManyErrorLogs fires on error, fatal, and panic logs with per-location granularity, making it easier to identify the exact…
otherFEATURE: vmui: show the range between the minimum and maximum values for each graph line in the legend. See. Thanks to @umer901 for contribution.
fixBUGFIX: all VictoriaMetrics components: hide values passed to -pushmetrics.header in startup logs, /metrics, and /flags, since they can contain sensitive HTTP headers such as Authorization and API keys. See.
fixBUGFIX: vmsingle, vmagent and vminsert in VictoriaMetrics cluster: fix insert requests getting stuck after another insert request times out, causing clients to time out while waiting for a response. See VictoriaLogs#1743.
fixBUGFIX: vmselect in VictoriaMetrics cluster: apply the le filter at /api/v1/status/metric_names_stats after merging request counters from all vmstorage nodes. Previously, the endpoint could report actively queried metrics as unused when the local request counter was zero on one of the nodes. See.…
fixBUGFIX: vmagent: now MDX discards cardinality_estimate and cardinality_churn_ratio metrics exposed by vmestimator service..
fixBUGFIX: vmselect in VictoriaMetrics cluster: allow access to /prometheus/api/v1/admin/tsdb/delete_series and graphite/tags/delSeries using only the corresponding -deleteAuthKey, even when -httpAuth. is also configured, without additionally requiring -httpAuth. credentials. See.
securitySECURITY: vmsingle, vmagent, vminsert and vmselect in VictoriaMetrics cluster: check HTTP Basic Auth credentials from the -httpAuth. command-line flags for cases that do not verify the corresponding *AuthKey command-line flag on its own. Previously, HTTP Basic Auth could be skipped for request…
featureFEATURE: all VictoriaMetrics components: add support for accepting HTTP requests over Unix domain socket at -httpListenAddr. For example, -httpListenAddr=unix:/run/victoria-metrics.sock. See VictoriaLogs#1618.
featureFEATURE: vmagent, vmsingle, vmstorage and vmselect in VictoriaMetrics cluster: expose the vm_app_prev_shutdown_unclean gauge. It is set to 1 when the previous process run didn't shut down cleanly. Added the UncleanShutdown alerting rule, which fires for 10 minutes after an unclean shutdown is…
featureFEATURE: vmui: show the selected time zone UTC offset next to the date/time controls and allow opening time zone settings from it. See.
otherFEATURE: vmsingle, vmagent, vmalert, and vmselect in VictoriaMetrics cluster: show how the default value is calculated for command-line flags which derive it from the number of available CPU cores. For example, -maxConcurrentInserts now prints (default 16 = 2*cgroup.AvailableCPUs()) in -help…
featureFEATURE: vmagent: introduce vmagent_kafka_consumer_broker_rtt_seconds and vmagent_kafka_consumer_fetch_queue_size metrics for Kafka consumer integration. These metrics help troubleshoot slow Kafka consumption. See.
fixBUGFIX: alerts: fix AlertingRuleResultsApproachingLimit and RecordingRuleResultsApproachingLimit alerting rules in vmalert alerting rules. The alerts could produce incorrect results or fail to fire when vmalert_group_rule_results_limit returned multiple samples per (group, file) label combination,…
fixBUGFIX: vmagent and vminsert in VictoriaMetrics cluster: fix infinite loop in the OpenTelemetry Firehose ingestion endpoint (/opentelemetry/api/v1/push) when receiving a malformed record with an incomplete varint in the data field. Previously this caused the goroutine to spin forever, permanently…
fixBUGFIX: vmalert-tool: reuse connections to -remoteWrite.url when writing the results of recording rules and alerts. Previously every series was sent over a new connection, which left a lot of sockets in TIME_WAIT state and could exhaust the ephemeral port range. The number of idle connections can…
fixBUGFIX: vmsingle and vmselect in VictoriaMetrics cluster: prevent process crash in sort_by_label_numeric() and sort_by_label_numeric_desc() when a label value contains a number with 309 or more digits. See.
fixBUGFIX: vmsingle and vmstorage in VictoriaMetrics cluster: fixes rare metrics metadata partial search response. See.
fixBUGFIX: vmctl: reuse connections in vm-native mode when --vm-concurrency exceeds 2. Previously the number of idle connections was limited to 2 per host, which was insufficient when --vm-concurrency was bigger than 2. See.
fixBUGFIX: vmsingle and vmselect in VictoriaMetrics cluster: properly re-use memory if query aggregation returns error. See.
fixBUGFIX: vmselect in VictoriaMetrics cluster: fail the query request directly when there is not enough disk space to store temporary search results. Previously, such queries could lead to vmselect crash. See.
fixBUGFIX: vmselect in VictoriaMetrics cluster: consistently re-use memory during storage blocks unpacking on parsing storage block error. See.
fixBUGFIX: vmalert: fix parsing of instant query sample timestamps from Prometheus-compatible data sources. Previously, decimal (e.g. 1786458420.123) or scientific-notation (e.g. 1.78645842E9) values were silently truncated to 0. See. Thanks to @missusk for the contribution.
fixBUGFIX: vmctl: properly release the export and import requests during migration requests fails in vm-native mode. Previously, failed export/import requests could have left hanging at the source or the destination. The fix is supposed to improve the resiliency of vmctl during long-running migrations.
fixBUGFIX: vmagent: include vm_account_id and vm_project_id labels in the instance identity used by monitoring data exchange filtering. Previously, when multitenant handlers were enabled, series from different tenants sharing the same job and instance labels could be misidentified as coming from a…
fixBUGFIX: vmsingle, vminsert in VictoriaMetrics cluster and vmagent: fix process crash at /api/v1/import/native endpoint when receiving a specially crafted ~45-byte request..
fixBUGFIX: vmsingle, vmagent and vmstorage in VictoriaMetrics cluster: allow access to /remotewrite-relabel-config, /api/v1/status/remotewrite-relabel-config, /remotewrite-url-relabel-config, /api/v1/status/remotewrite-url-relabel-config and /internal/log_new_series with only the corresponding…
securitySECURITY: upgrade Go builder from Go1.26.5 to Go1.26.6. See the list of issues addressed in Go1.26.6.
otherFEATURE: relabeling: reduce CPU usage up to 30% when matching relabeling rules with multiple if expressions containing exact metric names. Expressions for other metric names are now skipped before evaluating their remaining label filters. See. Thanks to @nevgeny for contribution.
featureFEATURE: vmagent and vmsingle: add support for linode_sd_configs for discovering scrape targets from Linode instances. See. Thanks to @cxdy for contribution.
otherFEATURE: vmalert: extend -replay.continueWithExecutionErr to also handle the 400 Bad Request response code, since it is used for Prometheus querying API requests when request parameters are missing or incorrect. See.
otherFEATURE: vmselect and vminsert in VictoriaMetrics cluster, and vmagent: set default value of -enableMultitenancyViaHeaders to true. This change enables support of multitenancy via headers for cluster and for vmagent by default, aligning VictoriaMetrics multitenancy behavior with multitenancy in…
featureFEATURE: vmui: add an option to customize the favicon color. This makes it easier to distinguish between different installations opened in multiple browser tabs. See.
fixBUGFIX: vmagent and vmsingle: avoid suggesting the unrelated -enableTCP6 command-line flag when scraping a target over a Unix domain socket fails. See. Thanks to @lwmacct for contribution.
fixBUGFIX: vmsingle and vminsert in VictoriaMetrics cluster: skip labels with empty name at /api/v1/import. Previously such a label replaced the metric name, so a series sent with "metric":{"name":"foo","":"bar"} was stored under the name bar. Other ingestion protocols already skip such labels. See.…
fixBUGFIX: vmsingle, vmselect in VictoriaMetrics cluster and vmctl: properly parse small fractional Unix timestamps in timestamp args such as start and end in /api/v1/query_range and --vm-native-filter-time-start and --vm-native-filter-time-end in vmctl. Previously, fractional Unix timestamps with…
fixBUGFIX: vmsingle, vmstorage and vmselect in VictoriaMetrics cluster: persist the previous working set cache during graceful shutdown when it is likely to contain the active working set. This prevents saving an empty or cold current cache right after split-mode cache rotation, which could otherwise…
fixBUGFIX: vmsingle and vmselect in VictoriaMetrics cluster: change the HTTP response code for Prometheus querying API requests from 422 Unprocessable Entity to 400 Bad Request when request parameters are missing or incorrect. See.
fixBUGFIX: vmui: respect the custom query step specified via g0.step_input when opening a URL. Previously, it could be reset to the automatically calculated step and potentially cause dashboards to freeze. See.
fixBUGFIX: vmagent and vmsingle: properly assign scrape target IP address at IPv6-only networks for docker_sd_configs. See.
fixBUGFIX: vmalert: rename vmalert_rule_group_results_limit back to vmalert_group_rule_results_limit. The metric was introduced in v1.147.0 but was accidentally given the wrong name. See.
fixBUGFIX: vmalert: properly update group-level eval_delay and eval_alignment for existing groups during runtime when config reload is triggered periodically or manually via /-/reload. Previously, these settings weren't updated after config reload during runtime. See.
fixBUGFIX: vmselect in VictoriaMetrics cluster: scale the default -search.maxConcurrentRequests with the number of available CPU cores instead of capping it at 16. See. Thanks to @Dhru1Tanna for contribution.
securitySECURITY: vmsingle and vmselect in VictoriaMetrics cluster: restrict /api/v1/admin/tsdb/delete_series, /tags/delSeries endpoints to POST method only to prevent some SSRF-based data deletion attacks. See.
featureFEATURE: dashboards: add Fsync avg duration panel to the Troubleshooting section of the single-node, cluster, and vmagent dashboards. This panel surfaces degradation of IO operation for faster incident triage. See.
featureFEATURE: vmagent: add name label identifying the corresponding -remoteWrite.url target to the vm_persistentqueue_* metrics exposed by persistent queue. See. Thanks to @tIGO for contribution.
featureFEATURE: vmagent: add -remoteWrite.obfuscateLabels flag for hashing values of the specified labels before sending metrics to the corresponding -remoteWrite.url. This allows sharing metrics with external systems while keeping sensitive label values hidden. See.
featureFEATURE: vmalert: add -replay.continueWithExecutionErr flag to allow continuing to replay other rules when a rule execution fails with a 422 response code, which can happen due to an expression syntax error or a resource limit being hit. See 11313.
featureFEATURE: vmalert: add template variable $interval to expose the alerting rule group's evaluation interval. This allows generating dashboard links with a lookback window relative to the rule's interval, for example &from={{ ($activeAt.Add (parseDurationTime (printf "-%s" .Interval))).UnixMilli…
featureFEATURE: vmbackupmanager and alerts: introduce vm_backup_last_success_at metric to track the last successful backup by type. Add alerting rules NoLatestBackupWithinLastDay, NoHourlyBackupWithinLastDay, NoDailyBackupWithinLast3Days, NoWeeklyBackupWithinLast14Days and NoMonthlyBackupWithinLast62Days…
otherFEATURE: vmctl: support Prometheus native histograms migration in remote read mode. Native histograms are converted into _count, _sum and _bucket series with vmrange labels in the same way as VictoriaMetrics converts native histograms received via Prometheus remote write protocol, except that for…
otherFEATURE: vminsert in VictoriaMetrics cluster: enable slowness-based re-routing by default. Previously, -disableRerouting defaulted to true, which limited ingestion throughput to the slowest vmstorage node. Now -disableRerouting defaults to false, so vminsert automatically routes data away from the…
otherFEATURE: vmui: persist the selected auto-refresh interval in the URL. See VictoriaLogs#1310.
fixBUGFIX: MetricsQL: properly drop data points filtered out by an inner comparison operation when its result is used on the right side of another comparison. Previously, queries like foo != (bar > 100) could return unexpected results because filtered-out data points are represented internally as…
fixBUGFIX: vmagent and vmsingle: ignore HTTP proxy environment variables when scraping targets over Unix domain sockets. See. Thanks to @lwmacct for contribution.
fixBUGFIX: vmalert: fixed the display of rule state badges on the Groups page in the web UI. See.
fixBUGFIX: vmbackupmanager: previously, vmbackupmanager was crashing on startup when it failed to restore backup state from remote storage, causing a crash loop. Now it logs the error and continues running, retrying the state restore before each scheduled backup. Added…
fixBUGFIX: stream aggregation: fix incorrect sum_samples_total results when enable_windows: true is set. See. Thanks to @beyond-infra for contribution.
breakingBUGFIX: vmsingle, vmselect in VictoriaMetrics cluster and vmctl: accept scientific notation with sub-second precision (e.g. 1.784144612388E9) for timestamp args such as start and end in /api/v1/query_range and --vm-native-filter-time-start and --vm-native-filter-time-end in vmctl. Previously,…