Prometheus 3.5.3 → 3.10.0
12 releases · 3 breaking changes · 4 security fixes · 0 advisories fixed
4security3breaking1deprecated48feature78fix
Breaking changes
- breaking Remove the feature flag and make sure that everything still works as intended. in 3.8.0
- breaking [BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389 in 3.7.3
- breaking [BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344 in 3.7.1
Security changes
- security [SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060 in 3.5.5
- security [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650 in 3.5.4
- security [SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934 in 3.5.4
- security [SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935 in 3.5.4
Deprecations
- deprecated [CHANGE] Remote-write: the following metrics are deprecated: in 3.7.0
Every release in between
feature[CHANGE] Alerting: Add alertmanager dimension to following metrics: prometheus_notifications_dropped_total, prometheus_notifications_queue_capacity, prometheus_notifications_queue_length. #16355
other[CHANGE] UI: Hide expanded alert annotations by default, enabling more information density on the /alerts page. #17611
feature[FEATURE] AWS SD: Add MSK Role. #17600
feature[FEATURE] PromQL: Add fill() / fill_left() / fill_right() binop modifiers for specifying default values for missing series. #17644
feature[FEATURE] Web: Add OpenAPI 3.2 specification for the HTTP API at /api/v1/openapi.yaml. #17825
feature[FEATURE] Dockerfile: Add distroless image variant using UID/GID 65532 and no VOLUME declaration. Busybox image remains default. #17876
feature[FEATURE] Web: Add on-demand wall time profiling under /debug/pprof/fgprof. #18027
feature[ENHANCEMENT] PromQL: Add more detail to histogram quantile monotonicity info annotations. #15578
other[ENHANCEMENT] Alerting: Independent alertmanager sendloops. #16355
feature[ENHANCEMENT] TSDB: Experimental support for early compaction of stale series in the memory with configurable threshold stale_series_compaction_threshold in the config file. #16929
feature[ENHANCEMENT] Service Discovery: Service discoveries are now removable from the Prometheus binary through the Go build tag remove_all_sd and individual service discoveries can be re-added with the build tags enable__sd. Users can build a custom Prometheus with only the necessary SDs for a smaller…
other[ENHANCEMENT] Promtool: Support promql syntax features promql-duration-expr and promql-extended-range-selectors. #17926
other[PERF] PromQL: Avoid unnecessary label extraction in PromQL functions. #17676
other[PERF] PromQL: Improve performance of regex matchers like .-.-.*. #17707
feature[PERF] OTLP: Add label caching for OTLP-to-Prometheus conversion to reduce allocations and improve latency. #17860
other[PERF] API: Compute /api/v1/targets/relabel_steps in a single pass instead of re-running relabeling for each prefix. #17969
other[PERF] tsdb: Optimize LabelValues intersection performance for matchers. #18069
fix[BUGFIX] PromQL: Prevent query strings containing only UTF-8 continuation bytes from crashing Prometheus. #17735
fix[BUGFIX] Web: Fix missing X-Prometheus-Stopping header for /-/ready endpoint in NotReady state. #17795
fix[BUGFIX] PromQL: Fix PromQL info() function returning empty results when filtering by a label that exists on both the input metric and target_info. #17817
fix[BUGFIX] TSDB: Fix a bug during exemplar buffer grow/shrink that could cause exemplars to be incorrectly discarded. #17863
fix[BUGFIX] UI: Fix broken graph display after page reload, due to broken Y axis min encoding/decoding. #17869
fix[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields (Labels, Histogram pointers, metadata strings) before returning buffers to pools. #17879
fix[BUGFIX] PromQL: info function: fix series without identifying labels not being returned. #17898
fix[BUGFIX] OTLP: Filter name from OTLP attributes to prevent duplicate labels. #17917
fix[BUGFIX] TSDB: Fix division by zero when computing stale series ratio with empty head. #17952
fix[BUGFIX] OTLP: Fix potential silent data loss for sum metrics. #17954
fix[BUGFIX] PromQL: Fix smoothed interpolation across counter resets. #17988
fix[BUGFIX] PromQL: Fix panic with @ modifier on empty ranges. #18020
fix[BUGFIX] PromQL: Fix avg_over_time for a single native histogram. #18058
fix[BUGFIX] Agent: fix crash shortly after startup from invalid type of object. #17802
fix[BUGFIX] Scraping: fix relabel keep/drop not working. #17807
other[CHANGE] Native Histograms are no longer experimental! Make the native-histogram feature flag a no-op. Use scrape_native_histograms config option instead. #17528
feature[CHANGE] API: Add maximum limit of 10,000 sets of statistics to TSDB status endpoint. #17647
feature[FEATURE] API: Add /api/v1/features for clients to understand which features are supported. #17427
feature[FEATURE] Promtool: Add start_timestamp field for unit tests. #17636
feature[FEATURE] Promtool: Add --format seriesjson option to tsdb dump to output just series labels in JSON format. #13409
feature[FEATURE] Add --storage.tsdb.delay-compact-file.path flag for better interoperability with Thanos. #17435
feature[FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. #17714
feature[ENHANCEMENT]: TSDB: add flag --storage.tsdb.block-reload-interval to configure TSDB Block Reload Interval. #16728
feature[ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. #17565
other[ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. #16834
feature[ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. #17374
feature[ENHANCEMENT] Notifications: Add a histogram prometheus_notifications_latency_histogram_seconds to complement the existing summary. #16637
feature[ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. #17483
feature[ENHANCEMENT] SD: add a config label with job name for most prometheus_sd_refresh metrics. #17138
feature[ENHANCEMENT] TSDB: New histogram prometheus_tsdb_sample_ooo_delta, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. #17477
other[ENHANCEMENT] Remote-read: Validate histograms received via remote-read. #17561
other[PERF] TSDB: Small optimizations to postings index. #17439
other[PERF] Scraping: Speed up relabelling of series. #17530
other[PERF] PromQL: Small optimisations in binary operators. #17524, #17519.
fix[BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in _total. #17682
fix[BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. #17602
fix[BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. #17605
fix[BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. #17678
fix[BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty ignoring() and non-empty grouping. #17643
fix[BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. #17479
fix[BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. #17668
fix[BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. #17708
fix[BUGFIX] PromQL: Fix collision in label_join and label_replace with non-overlapping series. #17703
fix[BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. #17161
fix[BUGFIX] PromQL: Ensure that rate/increase/delta of histograms results in a gauge histogram. #17608
fix[BUGFIX] PromQL: Do not panic while iterating over invalid histograms. #17559
fix[BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. #17533
fix[BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. #17561
fix[BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. #17546
fix[BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. #17472
fix[BUGFIX] remote: Fix Remote Write receiver, so it does not send wrong response headers for v1 flow and cause Prometheus senders to emit false partial error log and metrics. #17683
otherUpgrade to v3.8 and keep the feature flag. Everything should work as before.
otherAt your own pace, set scrape_native_histograms to true in all relevant scrape configs. (There is a global and a per-scrape-config version of scrape_native_histograms, allowing granular control if needed. It is a good idea to also set scrape_native_histograms explicitly to false where you do not…
breakingRemove the feature flag and make sure that everything still works as intended.
otherNow you are ready for an upgrade to the next release (v3.9).
other[CHANGE] Remote-write 2 (receiving): Update to 2.0-rc.4 spec. "created timestamp" (CT) is now called "start timestamp" (ST). #17411
other[CHANGE] TSDB: Native Histogram Custom Bounds with a NaN threshold are now rejected. #17287
other[FEATURE] OAuth2: support jwt-bearer grant-type (RFC7523 3.1). #17592
feature[FEATURE] Dockerfile: Add OpenContainers spec labels to Dockerfile. #16483
feature[FEATURE] SD: Add unified AWS service discovery for ec2, lightsail and ecs services. #17046
other[FEATURE] Native histograms are now a stable, but optional feature, use the scrape_native_histograms config setting. #17232 #17315
other[FEATURE] UI: Support anchored and smoothed keyword in promql editor. #17239
other[FEATURE] UI: Show detailed relabeling steps for each discovered target. #17337
feature[FEATURE] Alerting: Add urlQueryEscape to template functions. #17403
feature[FEATURE] Promtool: Add Remote-Write 2.0 support to promtool push metrics via the --protobuf_message flag. #17417
other[ENHANCEMENT] Clarify the docs about handling negative native histograms. #17249
feature[ENHANCEMENT] Mixin: Add static UID to the remote-write dashboard. #17256
feature[ENHANCEMENT] PromQL: Reconcile mismatched NHCB bounds in Add and Sub. #17278
feature[ENHANCEMENT] Alerting: Add "unknown" state for alerting rules that haven't been evaluated yet. #17282
feature[ENHANCEMENT] Scrape: Allow simultaneous use of classic histogram → NHCB conversion and zero-timestamp ingestion. #17305
feature[ENHANCEMENT] UI: Add smoothed/anchored in explain. #17334
other[ENHANCEMENT] OTLP: De-duplicate any target_info samples with the same timestamp for the same series. #17400
other[ENHANCEMENT] Document use_fips_sts_endpoint in sigv4 config sections. #17304
other[ENHANCEMENT] Document Prometheus Agent. #14519
other[PERF] PromQL: Speed up parsing of variadic functions. #17316
other[PERF] UI: Speed up alerts/rules/... pages by not rendering collapsed content. #17485
other[PERF] UI: Performance improvement when getting label name and values in promql editor. #17194
other[PERF] UI: Speed up /alerts for many firing alerts via virtual scrolling. #17254
fix[BUGFIX] PromQL: Fix slice indexing bug in info function on churning series. #17199
fix[BUGFIX] API: Reduce lock contention on /api/v1/targets. #17306
fix[BUGFIX] PromQL: Consistent handling of gauge vs. counter histograms in aggregations. #17312
fix[BUGFIX] TSDB: Allow NHCB with -Inf as the first custom value. #17320
fix[BUGFIX] UI: Fix duplicate loading of data from the API speed up rendering of some pages. #17357
fix[BUGFIX] Old UI: Fix createExpressionLink to correctly build /graph URLs so links from Alerts/Rules work again. #17365
fix[BUGFIX] PromQL: Avoid panic when parsing malformed info call. #17379
fix[BUGFIX] PromQL: Include histograms when enforcing sample_limit. #17390
fix[BUGFIX] Config: Fix panic if TLS CA file is absent. #17418
fix[BUGFIX] PromQL: Fix histogram_fraction for classic histograms and NHCB if lower bound is in the first bucket. #17424
breaking[BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389
fix[BUGFIX] Fix federation of some native histograms. #17299 #17409
fix[BUGFIX] promtool: check config would fail when --lint=none flag was set. #17399 #17414
fix[BUGFIX] Remote-write: fix a deadlock in the queue resharding logic that can lead to suboptimal queue behavior. #17412
fix[BUGFIX] AWS SD: Fix AWS SDK v2 credentials handling for EC2 and Lightsail discovery. #17355
fix[BUGFIX] AWS SD: Load AWS region from IMDS when not set. #17376
fix[BUGFIX] Relabeling: Fix labelmap action validation with the legacy metric name validation scheme. #17372
fix[BUGFIX] PromQL: Fix parsing failure when anchored and smoothed are used as metric names and label names. #17353
fix[BUGFIX] PromQL: Fix formatting of range vector selectors with smoothed/anchored modifier. #17354
breaking[BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344
deprecated[CHANGE] Remote-write: the following metrics are deprecated:
otherprometheus_remote_storage_samples_in_total, use prometheus_wal_watcher_records_read_total{type="samples"} and prometheus_remote_storage_samples_dropped_total instead
otherprometheus_remote_storage_histograms_in_total, use prometheus_wal_watcher_records_read_total{type=~".*histogram_samples"} and prometheus_remote_storage_histograms_dropped_total instead
otherprometheus_remote_storage_exemplars_in_total, use prometheus_wal_watcher_records_read_total{type="exemplars"} and prometheus_remote_storage_exemplars_dropped_total instead
otherprometheus_remote_storage_highest_timestamp_in_seconds, use the more accurate prometheus_remote_storage_queue_highest_timestamp_seconds instead in dashboards and alerts to properly account for relabeling and for more accuracy. #17065
feature[FEATURE] PromQL: Add support for experimental anchored and smoothed rate behind feature flag promql-extended-range-selectors. #16457
feature[FEATURE] Federation: Add support for native histograms with custom buckets (NHCB). #17215
feature[FEATURE] PromQL: Add first_over_time(...) and ts_of_first_over_time(...) behind feature flag experimental-promql-functions. #16963 #17021
feature[FEATURE] Remote-write: Add support for Azure Workload Identity as an authentication method for the receiver. #16788
feature[FEATURE] Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the type-and-unit-labels feature flag is enabled. #17033
other[FEATURE] OTLP: Write start time of metrics as created time zero samples into TSDB when created-timestamp-zero-ingestion feature flag is enabled. #16951
feature[ENHANCEMENT] PromQL: Add warn-level annotations for counter reset conflicts in certain histogram operations. #17051 #17094
feature[ENHANCEMENT] UI: Add scrape interval and scrape timeout to targets page. #17158
other[ENHANCEMENT] TSDB: Reduce the resolution of native histograms read from chunks or remote read if the schema is exponential. #17213
feature[ENHANCEMENT] Remote write: Add logging for unexpected metadata in sample batches, when metadata entries are found in samples-only batches. #17034 #17082
other[ENHANCEMENT] Rules: Support concurrent evaluation for rules querying ALERTS and ALERTS_FOR_STATE. #17064
feature[ENHANCEMENT] TSDB: Add logs to improve visibility into internal operations. #17074
other[PERF] OTLP: Write directly to TSDB instead of passing through a Remote-Write adapter when receiving OTLP metrics. #16951
other[PERF] OTLP: Reduce number of logs emitted from OTLP endpoint. No need to log duplicate sample errors. #17201
other[PERF] PromQL: Move more work to preprocessing step. #16896
other[PERF] PromQL: Reduce allocations when walking the syntax tree. #16593
other[PERF] TSDB: Optimize appender creation, slightly speeding up startup. #16922
other[PERF] TSDB: Improve speed of querying a series with multiple matchers. #13971
fix[BUGFIX] Alerting: Mutating alerts relabeling (using replace actions, etc.) within a alertmanager_config.alert_relabel_configs block is now scoped correctly and no longer yields altered alerts to subsequent blocks. #17063
fix[BUGFIX] Config: Infer valid escaping scheme when scrape config validation scheme is set. #16923
fix[BUGFIX] TSDB: Correctly handle appending mixed-typed samples to the same series. #17071 #17241 #17290 #17295 #17296
fix[BUGFIX] Remote-write: Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0, log warning. #17146
fix[BUGFIX] TSDB: Fix metadata entries handling on metadata-wal-records experimental feature for native histograms with custom buckets (NHCB) in protobuf scraping. #17156
fix[BUGFIX] TSDB: Ignore Native Histograms with invalid schemas during WAL/WBL replay. #17214
fix[BUGFIX] PromQL: Avoid empty metric names in annotations for histogram_quantile(). #16794
fix[BUGFIX] PromQL: Correct inaccurate character positions in errors for some aggregate expressions. #16996 #17031
fix[BUGFIX] PromQL: Fix info() function on churning series. #17135
fix[BUGFIX] PromQL: Set native histogram to gauge type when subtracting or multiplying/dividing with negative factors. #17004
fix[BUGFIX] TSDB: Reject unsupported native histogram schemas when attempting to append to TSDB. For scrape and remote-write implement reducing the resolution to fit the maximum if the schema is within the -9 to 52. #17189
fix[BUGFIX] Remote-write: Fix HTTP handler to return after writing error response for invalid compression. #17050
fix[BUGFIX] Remote-write: Return HTTP error 400 instead of 5xx for wrongly formatted Native Histograms. #17210
fix[BUGFIX] Scrape: Prevent staleness markers from generating unnecessary series. #16429
fix[BUGFIX] TSDB: Avoid misleading Failed to calculate size of \"wal\" dir error logs during WAL clean-up. #17006
fix[BUGFIX] TSDB: Prevent erroneously dropping series records during WAL checkpoints. #17029
fix[BUGFIX] UI: Fix redirect to path of -web.external-url if -web.route-prefix is configured. #17240
+1 more item in the release notesfeature[FEATURE] PromQL: Add step(), and min() and max() on durations, behind feature flag promql-duration-expr. #16777
feature[FEATURE] API: Add a /v1/status/tsdb/blocks endpoint exposing metadata about loaded blocks. #16695
feature[FEATURE] Templates: Add toDuration() and now() functions. #16619
feature[ENHANCEMENT] Discovery: Add support for attaching namespace metadata to targets. #16831
feature[ENHANCEMENT] OTLP: Support new UnderscoreEscapingWithoutSuffixes strategy via otlp.translation_strategy. #16849
other[ENHANCEMENT] OTLP: Support including scope metadata as metric labels via otlp.promote_scope_metadata. #16878
feature[ENHANCEMENT] OTLP: Add type and unit labels when feature flag type-and-unit-labels is enabled. #16630
other[ENHANCEMENT] Tracing: Send the traceparent HTTP header during scrapes. #16425
feature[ENHANCEMENT] UI: Add option to disable info and warning query messages under Query page settings. #16901
other[ENHANCEMENT] UI: Improve metadata handling for _count/_sum/_bucket suffixes. #16910
other[ENHANCEMENT] TSDB: Track stale series in the Head block via the prometheus_tsdb_head_stale_series metric. #16925
other[PERF] PromQL: Improve performance due to internal optimizations. #16797
fix[BUGFIX] Config: Fix "unknown global name escaping method" error messages produced during config validation. #16801
fix[BUGFIX] Discovery: Fix race condition during shutdown. #16820
fix[BUGFIX] OTLP: Generate target_info samples between the earliest and latest samples per resource. #16737
fix[BUGFIX] PromQL: Fail when NaN is passed as parameter to topk(), bottomk(), limitk() and limit_ratio(). #16725
fix[BUGFIX] PromQL: Fix extrapolation for native counter histograms. #16828
fix[BUGFIX] PromQL: Reduce numerical errors by disabling some optimizations. #16895
fix[BUGFIX] PromQL: Fix inconsistencies when using native histograms in subqueries. #16879
fix[BUGFIX] PromQL: Fix inconsistent annotations for rate() and increase() on histograms when feature flag type-and-unit-labels is enabled. #16915
fix[BUGFIX] Scraping: Fix memory corruption in slicelabels builds. #16946
fix[BUGFIX] TSDB: Fix panic on append when feature flag created-timestamp-zero-ingestion is enabled. #16332
fix[BUGFIX] TSDB: Fix panic on append for native histograms with empty buckets. #16893
security[SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060
security[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650
security[SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934
security[SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935
other[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18792