Prometheus 3.2.1 → 3.15.1
35 releases · 12 breaking changes · 35 security fixes · 2 advisories fixed
35security12breaking8deprecated129feature231fix
Vulnerabilities fixed by this upgrade
- 7.5 CVE-2026-42151 Prometheus Azure AD remote write OAuth client secret exposed via config API
- 7.5 CVE-2026-42154 Remote read endpoint allows denial of service via crafted snappy payload
Breaking changes
- breaking [BUGFIX] Discovery: delete the stale prometheus_sd_last_update_timestamp_seconds series for a config that is removed on reload. #19131 in 3.15.0
- breaking [BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326 in 3.15.0
- breaking [BUGFIX] Rules: Clean up stale rule_group_last_rule_duration_sum_seconds and rule_group_last_restore_duration_seconds series when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing /metrics cardinality over time. #19107 in 3.14.0
- breaking [BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. #18849 in 3.13.0
- breaking [BUGFIX] prometheus_sd_refresh* and prometheus_sd_discovered_targets metrics for specific scrape jobs are deleted when the scrape job is removed. #17614 in 3.12.0
- breaking [ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. #17598 in 3.11.0
- breaking [BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. #18200 in 3.11.0
- breaking Remove the feature flag and make sure that everything still works as intended. in 3.8.0
- breaking [BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389 in 3.7.3
- breaking [BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344 in 3.7.1
- breaking [BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895 in 3.3.0
- breaking [BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060 in 3.3.0
Security changes
- security [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys. in 3.15.1
- security [SECURITY] Build with Go 1.27.3 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template. in 3.15.1
- security [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617). in 3.15.1
- security [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys. in 3.13.5
- security [SECURITY] Build with Go 1.26.10 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template. in 3.13.5
- security [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617). in 3.13.5
- security [SECURITY] Bump OpenTelemetry to v1.45.0 to fix exporter endpoint URLs leaking into info logs (GO-2026-6505). in 3.13.5
- security [SECURITY] Bump google.golang.org/grpc to v1.83.1 to fix HTTP/2 DATA frame fragmentation memory exhaustion (GO-2026-6348). #19834 in 3.13.4
- security [SECURITY] UI: Update vulnerable npm dependencies. #19834 in 3.13.4
- security [SECURITY] Bump github.com/klauspost/compress to v1.18.7 (GO-2026-5841) and golang.org/x/crypto to v0.55.0 (GO-2026-6303). in 3.13.3
- security [SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama in 3.13.2
- security [SECURITY] UI: Bump sanitize-html to fix a cross-site scripting vulnerability (CVE-2026-44990). #18697 in 3.13.0
- security [CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673… in 3.13.0
- security [SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to @hibrian827 for reporting it. #18642 in 3.12.0
- security [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18649 in 3.12.0
- security Shadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability. in 3.11.3
- security Brett Gervasoni for the AzureAD OAuth client_secret vulnerability. in 3.11.3
- security @iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability. in 3.11.3
- security [SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18590 in 3.11.3
- security [SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18584 in 3.11.3
- security [SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18588 in 3.11.3
- security [SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18506 in 3.11.2
- security [ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. #18347 in 3.11.0
- security [SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060 in 3.5.5
- security [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650 in 3.5.4
- security [SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934 in 3.5.4
- security [SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935 in 3.5.4
- security Shadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability. in 3.5.3
- security Brett Gervasoni for the AzureAD OAuth client_secret vulnerability. in 3.5.3
- security @iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability. in 3.5.3
- security [SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18587 in 3.5.3
- security [SECURITY] Remote-Write: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. #18591 in 3.5.3
- security [SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18585 in 3.5.3
- security [SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18589 in 3.5.3
- security [SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18507 in 3.5.2
Deprecations
- deprecated [CHANGE] Logging: Deprecate --log.level; use runtime.log_level configuration to supply the default level. #19511 in 3.15.0
- deprecated [ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. --enable-feature=xor2-encoding is deprecated; use storage.tsdb.chunk_encoding.floats: xor2 instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461 in 3.15.0
- deprecated [BUGFIX] discovery/kubernetes: Populate __meta_kubernetes_service_loadbalancer_ip from status.loadBalancer.ingress, falling back to deprecated spec.loadBalancerIP. #19404 in 3.15.0
- deprecated [CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 in 3.14.0
- deprecated [CHANGE] Hetzner SD: The meta_hetzner_datacenter label is deprecated for the role robot but kept for backward compatibility, use the meta_hetzner_robot_datacenter label instead. For the role hcloud, the label is deprecated and will stop working after the 1 July 2026. #17850 in 3.11.0
- deprecated [CHANGE] Hetzner SD: The meta_hetzner_hcloud_datacenter_location and meta_hetzner_hcloud_datacenter_location_network_zone labels are deprecated, use the meta_hetzner_hcloud_location and meta_hetzner_hcloud_location_network_zone labels instead. #17850 in 3.11.0
- deprecated [ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. #17829 in 3.11.0
- deprecated [CHANGE] Remote-write: the following metrics are deprecated: in 3.7.0
Every release in between
security[SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
security[SECURITY] Build with Go 1.27.3 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template.
security[SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
fix[CHANGE] PromQL: A range query whose end was not aligned to step caused subqueries inside it to evaluate past the parent's last actual step, inflating peakSamples in the query stats and against the query.max-samples limit, and wasting storage I/O reading samples that were never used in the result.…
deprecated[CHANGE] Logging: Deprecate --log.level; use runtime.log_level configuration to supply the default level. #19511
feature[FEATURE] Configuration: Allow changing the process log level through runtime.log_level on configuration reload. #19511
feature[FEATURE] Prometheus: Add --auto-gomemlimit.refresh-interval flag to periodically re-detect the container or system memory limit and update GOMEMLIMIT at runtime. #18843
feature[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. #12024. #18091
feature[FEATURE] scrape: Implement OM2.0 scrape format. #18606
feature[ENHANCEMENT] Docker SD: Add labels meta_docker_container_image and meta_docker_container_image_id. #19386
feature[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. #19500
deprecated[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. --enable-feature=xor2-encoding is deprecated; use storage.tsdb.chunk_encoding.floats: xor2 instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461
feature[ENHANCEMENT] TSDB: add prometheus_tsdb_head_appenders_created_total metric. #19411
feature[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. #19410
feature[ENHANCEMENT] storage/remote: Add undocumented failed_request_logging config field to debug log remote write V2 requests on send errors. #19249
feature[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. #18049
feature[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. #19332
feature[ENHANCEMENT] TSDB: Add prometheus_tsdb_head_series_pending_commit_underflow_total to report pending-sample reservation underflows. #19470
fix[BUGFIX] PromQL: Fix info() enrichment for composite expressions with mixed @/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. #19387
fix[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. #19557
fix[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. #19413
fix[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. #19272
fix[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. #19470
fix[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. #19470
fix[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. #19664
fix[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. #19460
fix[BUGFIX] TSDB: fix default block reload interval for custom options. #19368
fix[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. #19194
fix[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. #19435
fix[BUGFIX] AWS SD: Reject non-positive request_concurrency instead of hanging service discovery indefinitely. #19524
fix[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. #19302
fix[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. #19324
breaking[BUGFIX] Discovery: delete the stale prometheus_sd_last_update_timestamp_seconds series for a config that is removed on reload. #19131
fix[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. #19661
fix[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. #19438
fix[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. #19218
fix[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. #19444
fix[BUGFIX] Native histograms: DetectReset no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram rate()/increase() undercount. #19367
fix[BUGFIX] Never skip histogram buckets for histogram_stddev and histogram_stdvar functions. #19521
fix[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. #19343
fix[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. .\|(foo)\|.). #19516
breaking[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326
deprecated[BUGFIX] discovery/kubernetes: Populate __meta_kubernetes_service_loadbalancer_ip from status.loadBalancer.ingress, falling back to deprecated spec.loadBalancerIP. #19404
+40 more items in the release notesdeprecated[CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124
other[CHANGE] API: /api/v1/status/config now correctly shows separator: "" and replacement: "" in relabel configs when explicitly set to empty, instead of omitting them. #18653
other[CHANGE] Discovery/Hetzner: Drop the __meta_hetzner_datacenter label for hcloud targets, following its removal from the Hetzner Cloud API. #19269
other[CHANGE] PromQL: Enable duration expressions by default. The promql-duration-expr feature flag is now a no-op. #19033
other[CHANGE] PromQL: Promote first_over_time to stable. It no longer requires the promql-experimental-functions feature flag. #19093
feature[FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (oci_sd_configs). #18919
feature[FEATURE] PromQL: Add experimental start_timestamp(instant-vector) function returning the start timestamp of each sample in the given vector. Requires the use-start-timestamps feature flag. #19089
feature[FEATURE] PromQL: Allow rate() and increase() to use start timestamps as an alternative for rate extrapolation. Hidden behind the use-start-timestamps feature flag. #18619
feature[FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the histograms-st-encoding feature flag. #18609
other[ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g. k8s.pod.name and k8s_pod_name both become k8s_pod_name), and expose the prometheus_api_otlp_translation_warnings_total counter labelled by category to track such warnings.…
feature[ENHANCEMENT] Promtool: Add --remote-write.path flag to push metrics for backends that use a non-default remote-write endpoint. #19086
other[ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903
feature[ENHANCEMENT] TSDB: Add prometheus_tsdb_head_native_histogram_series and prometheus_tsdb_head_native_histogram_buckets gauges tracking the number of native histogram series and buckets in the head. #19170
feature[ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (step(), range(), min_of(), max_of()) in range selectors and subqueries. #18625
feature[ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706
other[ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181
other[PERF] Speed up regex label matchers matching a set of literal values (e.g. {job=~"foo|bar|baz"}). #18833
other[PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470
other[PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143
fix[BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
fix[BUGFIX] Discovery/AWS: Stop promtool check config from making AWS metadata service (IMDS) network calls when the region field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037
fix[BUGFIX] Discovery/Docker: Set a request timeout for docker_sd and dockerswarm_sd on unix, npipe, and tcp hosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237
fix[BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
fix[BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778
fix[BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
fix[BUGFIX] PromQL: Fix mad_over_time returning 0 instead of NaN when the range contains a NaN sample. #19040
fix[BUGFIX] Promtool: Accept --enable-feature=promql-binop-fill-modifiers in check rules, which previously rejected valid fill()/fill_left()/fill_right() expressions. #19153
fix[BUGFIX] Remote write: Respect the AZURE_FEDERATED_TOKEN_FILE environment variable for workload identity authentication instead of hardcoding the token file path. #18973
breaking[BUGFIX] Rules: Clean up stale rule_group_last_rule_duration_sum_seconds and rule_group_last_restore_duration_seconds series when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing /metrics cardinality over time. #19107
fix[BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
fix[BUGFIX] TSDB: Fix silent data loss and potential crash loop when stale_series_compaction_threshold is used in the config file. #19016
fix[BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted. #19016
fix[BUGFIX] TSDB: Fix prometheus_tsdb_head_stale_series over-counting and early eviction of series that change between float, integer histogram, and float histogram sample types. #19183
fix[BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
fix[BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
fix[BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
fix[BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart. #19202
fix[BUGFIX] TSDB: Surface query errors that were previously silently discarded. #19120
fix[BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with --enable-feature=xor2-encoding could silently revert to XOR after compaction. #19145
fix[BUGFIX] UI: Show the delete-series form on the TSDB Status page when --web.enable-admin-api is enabled. #19025
+2 more items in the release notessecurity[SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
security[SECURITY] Build with Go 1.26.10 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template.
security[SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
security[SECURITY] Bump OpenTelemetry to v1.45.0 to fix exporter endpoint URLs leaking into info logs (GO-2026-6505).
other[CHANGE] Building Prometheus now requires Go 1.26 or later.
security[SECURITY] Bump google.golang.org/grpc to v1.83.1 to fix HTTP/2 DATA frame fragmentation memory exhaustion (GO-2026-6348). #19834
security[SECURITY] UI: Update vulnerable npm dependencies. #19834
fix[BUGFIX] Agent: Ignore unknown WAL record types to allow rolling back from newer versions. #19814
fix[BUGFIX] Federation: Fix corruption of float native histograms. #19679
fix[BUGFIX] Scrape: Fix failing scrapes of protobuf float histograms with zero sample, zero and bucket counts, such as the result of recording rate(x[1m]) over a constant histogram. #19682
security[SECURITY] Bump github.com/klauspost/compress to v1.18.7 (GO-2026-5841) and golang.org/x/crypto to v0.55.0 (GO-2026-6303).
fix[BUGFIX] Docker SD: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
fix[BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
fix[BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
fix[BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
fix[BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
fix[BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
fix[BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
security[SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama
fix[BUGFIX] PromQL: Preallocate the active query tracker file to avoid SIGBUS crashes when the data disk is full. #19289 by @akshajrawat
fix[BUGFIX] TSDB: Fix the head-chunk cache returning samples from the wrong chunk, or spurious not-found errors, to range queries after head-chunk truncation. #19134
security[SECURITY] UI: Bump sanitize-html to fix a cross-site scripting vulnerability (CVE-2026-44990). #18697
other[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at /assets/third-party-licenses.txt, replacing the npm_licenses.tar.bz2 archive previously shipped in release tarballs and container images. #18997
other[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. #18927
security[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673…
other[CHANGE] promtool: Relative file paths in the file passed to --http.config.file are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. #18949
other[CHANGE] PromQL: Rename the min() and max() duration-expression functions (experimental feature flag experimental-duration-expr) to min_of() and max_of() to avoid confusion with the min and max aggregate operators. #18687
feature[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. #18573
feature[FEATURE] Discovery/AWS: Add ability to filter RDS instances. #18859
feature[FEATURE] PromQL: Add min_of(a, b) and max_of(a, b) scalar experimental functions, returning the smaller or larger of two scalar values. #18687
feature[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. #18564
feature[FEATURE] PromQL: Expose per-query samplesRead (and samplesReadPerStep with stats=all and the promql-per-step-stats feature flag) in the query stats response, and add the prometheus_engine_query_samples_read_total engine counter. samplesRead reflects storage I/O distinct from…
feature[FEATURE] Scrape: Add convert_classic_histograms_to_nhcb internal label to allow per-target override of convert_classic_histograms_to_nhcb scrape configuration via relabeling. #18840
feature[FEATURE] TSDB: Add storage.tsdb.chunk_encoding.floats configuration field to select float chunk encoding (xor or xor2) at runtime, independently of the --enable-feature=xor2-encoding flag. #18769
feature[FEATURE] remote_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. #18217
feature[FEATURE] Scrape: Add always_scrape_classic_histograms and scrape_native_histograms internal labels to allow per-target override of the always_scrape_classic_histograms and scrape_native_histograms scrape configuration via relabeling. #18929
other[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18791
other[ENHANCEMENT] PromQL: Prettify fill_left(x) fill_right(x) as fill(x) when both fill values are equal. #18851
other[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. #18894
feature[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to ~2x faster). #18540
other[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by ~12-15% in benchmarks. #18699
other[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (--enable-feature=created-timestamp-zero-ingestion). #18813
fix[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. #18845
fix[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. #18629
fix[BUGFIX] PromQL: A range query whose end was not aligned to step caused subqueries inside it to evaluate past the parent's last actual step, inflating peakSamples in the query stats and against the query.max-samples limit, and wasting storage I/O reading samples that were never used in the result.…
fix[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an @ modifier (e.g. predict_linear(metric[60s] @ T, X)) silently under-counted totalQueryableSamples for steps after step 0. #18081
fix[BUGFIX] PromQL: Fix fill_left/fill_right producing missing samples in range queries when using group_left/group_right. #18850
fix[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. #18906
fix[BUGFIX] PromQL: Fix panic on 1[5m] smoothed and similar expressions when extended range selectors are enabled. #18764
fix[BUGFIX] PromQL: Fix panic when a smoothed instant vector selector produces no samples for a series. #18943
fix[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. foo offset -(5)). #18768
fix[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces {}. Via prometheus/common v0.69.0. #18949
fix[BUGFIX] Promtool: Fix check healthy and check ready when --url ends with a trailing slash. #18854
fix[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. #18733
fix[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy private_ip or public_ip field, but do have private NICs attached. #18772
fix[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). #18838
fix[BUGFIX] UI: Escape label values offered by PromQL autocomplete. #18658
fix[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. #18739
fix[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. #18847
breaking[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. #18849
featureSecurity: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery.
featurePromQL & Metadata: Several features and bug fixes related to the experimental "start timestamps" support, including updates to rate(), irate(), increase(), and resets(). New experimental functions start(), end(), range(), and step() are introduced.
featureTSDB Performance: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage.
featureService Discovery: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support).
featureUI: Added a web interface for deleting time series and cleaning tombstones.
security[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to @hibrian827 for reporting it. #18642
security[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18649
feature[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used st-storage flag is enabled. #18221
feature[FEATURE] API: Add /api/v1/status/self_metrics endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. #18411
feature[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery #18287
feature[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target #18321
feature[FEATURE] Discovery: Add Outscale VM service discovery (outscale_sd_configs) for discovering scrape targets from the Outscale Cloud API. #18139
feature[FEATURE] PromQL: Add start(), end(), range(), and step() experimental functions #17877
feature[FEATURE] TSDB/Agent: Add CheckpointFromInMemorySeries option to agent.DB that enables checkpoint based on in-memory series. #17948
feature[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. #18390
feature[FEATURE] Scrape: Added a feature flag st-synthesis which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. #18279
feature[ENHANCEMENT] AWS SD: Add optional external_id field to ECS/MSK/RDS/Elasticache. #18579
feature[ENHANCEMENT] AWS SD: Add optional external_id field. #17171
feature[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. #18187
feature[ENHANCEMENT] Promtool: Add --header flag to query instant command, matching existing query range behaviour. #18418
feature[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. #16088
fix[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. #18568
fix[BUGFIX] PromQL: Fix info() function incorrectly handling negated name matchers #17932
fix[BUGFIX] API: Return duration expressions in /parse_ast. #18624
fix[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec #18305
fix[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 #18567
fix[BUGFIX] Config: Validate remote_write queue_config fields at load time to prevent runtime panic and silent misconfiguration. #18209
fix[BUGFIX] Discovery/Consul: Add health_filter for Health API filtering, fixing breakage when using Catalog-only fields like ServiceTags in filter. #18479 #18499
fix[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. #18408
fix[BUGFIX] PromQL: Fix smoothed rate/increase returning zero instead of no result when all data falls strictly after the query range. #18523
fix[BUGFIX] PromQL: Fix metric name not being dropped when last_over_time or first_over_time is applied to subqueries containing name-dropping functions like abs(). #18409
fix[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. #18660
fix[BUGFIX] PromQL: Fix parsing of range() keyword in duration expressions such as foo[5m+range()]. #18623
fix[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the @ modifier is used. #18531
fix[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. #18639
fix[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. #18414
fix[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. #18382
fix[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. #18421
fix[BUGFIX] TSDB: Allow retention percentage with new data path. #18628
breaking[BUGFIX] prometheus_sd_refresh* and prometheus_sd_discovered_targets metrics for specific scrape jobs are deleted when the scrape job is removed. #17614
+18 more items in the release notessecurityShadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability.
securityBrett Gervasoni for the AzureAD OAuth client_secret vulnerability.
security@iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability.
security[SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18590
security[SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18584
security[SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18588
security[SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18506
feature[ENHANCEMENT] Consul SD: Introduce health_filter field for Health API filtering. #18499
fix[BUGFIX] Consul SD: Fix filter parameter being incorrectly applied to the Health API. #18499
fix[BUGFIX] Tracing: Fix startup failure for OTLP HTTP tracing with insecure: true. #18469
deprecated[CHANGE] Hetzner SD: The meta_hetzner_datacenter label is deprecated for the role robot but kept for backward compatibility, use the meta_hetzner_robot_datacenter label instead. For the role hcloud, the label is deprecated and will stop working after the 1 July 2026. #17850
deprecated[CHANGE] Hetzner SD: The meta_hetzner_hcloud_datacenter_location and meta_hetzner_hcloud_datacenter_location_network_zone labels are deprecated, use the meta_hetzner_hcloud_location and meta_hetzner_hcloud_location_network_zone labels instead. #17850
feature[FEATURE] AWS SD: Add Elasticache Role. #18099
feature[FEATURE] AWS SD: Add RDS Role. #18206
feature[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. #17207
feature[FEATURE] Discovery: Introduce prometheus_sd_last_update_timestamp_seconds metric to track the last time a service discovery update was sent to consumers. #18194
feature[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. #18006
feature[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: meta_kubernetes_pod_deployment_name, meta_kubernetes_pod_cronjob_name and __meta_kubernetes_pod_job_name, respectively. #17774
feature[FEATURE] PromQL: Add / operators for trimming observations from native histograms. #17904
feature[FEATURE] PromQL: Add experimental histogram_quantiles variadic function for computing multiple quantiles at once. #17285
feature[FEATURE] TSDB: Add storage.tsdb.retention.percentage configuration to configure the maximum percent of disk usable for TSDB storage. #18080
feature[FEATURE] TSDB: Add an experimental st-storage feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. #18062
feature[FEATURE] TSDB: Add an experimental xor2-encoding feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. #18062
feature[ENHANCEMENT] HTTP client: Add AWS external_id support for sigv4. #17916
deprecated[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. #17829
breaking[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. #17598
feature[ENHANCEMENT] UI: Add autocomplete support for experimental first_over_time and ts_of_first_over_time PromQL functions. #18318
security[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. #18347
fix[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using KahanAdd. #18252
fix[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured endpoint option, a regression from the AWS SDK v2 migration. #18133
fix[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. #18133
fix[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. #17538
fix[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. #18244
fix[BUGFIX] Azure SD: Fix system-assigned managed identity not working when client_id is empty. #18323
fix[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. #17349
fix[BUGFIX] Kubernetes SD: Fix duplicate targets generated by *DualStack EndpointSlices policies. #18192
fix[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. #18084
fix[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. #18056
fix[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. #18263
fix[BUGFIX] Remote write: Fix prometheus_remote_storage_sent_batch_duration_seconds measuring before the request was sent. #18214
fix[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. #18375
fix[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. #18229
fix[BUGFIX] TSDB: Fail early when use-uncached-io feature flag is set on unsupported environments. #18219
breaking[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. #18200
fix[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. #17895
fix[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. #18306
fix[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. #18200
fix[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. #18189
fix[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. #18228
fix[BUGFIX] UI: Update retention display in runtime info when config is reloaded. #18200
+5 more items in the release notesfeature[CHANGE] Alerting: Add alertmanager dimension to following metrics: prometheus_notifications_dropped_total, prometheus_notifications_queue_capacity, prometheus_notifications_queue_length. #16355
other[CHANGE] UI: Hide expanded alert annotations by default, enabling more information density on the /alerts page. #17611
feature[FEATURE] AWS SD: Add MSK Role. #17600
feature[FEATURE] PromQL: Add fill() / fill_left() / fill_right() binop modifiers for specifying default values for missing series. #17644
feature[FEATURE] Web: Add OpenAPI 3.2 specification for the HTTP API at /api/v1/openapi.yaml. #17825
feature[FEATURE] Dockerfile: Add distroless image variant using UID/GID 65532 and no VOLUME declaration. Busybox image remains default. #17876
feature[FEATURE] Web: Add on-demand wall time profiling under /debug/pprof/fgprof. #18027
feature[ENHANCEMENT] PromQL: Add more detail to histogram quantile monotonicity info annotations. #15578
other[ENHANCEMENT] Alerting: Independent alertmanager sendloops. #16355
feature[ENHANCEMENT] TSDB: Experimental support for early compaction of stale series in the memory with configurable threshold stale_series_compaction_threshold in the config file. #16929
feature[ENHANCEMENT] Service Discovery: Service discoveries are now removable from the Prometheus binary through the Go build tag remove_all_sd and individual service discoveries can be re-added with the build tags enable__sd. Users can build a custom Prometheus with only the necessary SDs for a smaller…
other[ENHANCEMENT] Promtool: Support promql syntax features promql-duration-expr and promql-extended-range-selectors. #17926
other[PERF] PromQL: Avoid unnecessary label extraction in PromQL functions. #17676
other[PERF] PromQL: Improve performance of regex matchers like .-.-.*. #17707
feature[PERF] OTLP: Add label caching for OTLP-to-Prometheus conversion to reduce allocations and improve latency. #17860
other[PERF] API: Compute /api/v1/targets/relabel_steps in a single pass instead of re-running relabeling for each prefix. #17969
other[PERF] tsdb: Optimize LabelValues intersection performance for matchers. #18069
fix[BUGFIX] PromQL: Prevent query strings containing only UTF-8 continuation bytes from crashing Prometheus. #17735
fix[BUGFIX] Web: Fix missing X-Prometheus-Stopping header for /-/ready endpoint in NotReady state. #17795
fix[BUGFIX] PromQL: Fix PromQL info() function returning empty results when filtering by a label that exists on both the input metric and target_info. #17817
fix[BUGFIX] TSDB: Fix a bug during exemplar buffer grow/shrink that could cause exemplars to be incorrectly discarded. #17863
fix[BUGFIX] UI: Fix broken graph display after page reload, due to broken Y axis min encoding/decoding. #17869
fix[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields (Labels, Histogram pointers, metadata strings) before returning buffers to pools. #17879
fix[BUGFIX] PromQL: info function: fix series without identifying labels not being returned. #17898
fix[BUGFIX] OTLP: Filter name from OTLP attributes to prevent duplicate labels. #17917
fix[BUGFIX] TSDB: Fix division by zero when computing stale series ratio with empty head. #17952
fix[BUGFIX] OTLP: Fix potential silent data loss for sum metrics. #17954
fix[BUGFIX] PromQL: Fix smoothed interpolation across counter resets. #17988
fix[BUGFIX] PromQL: Fix panic with @ modifier on empty ranges. #18020
fix[BUGFIX] PromQL: Fix avg_over_time for a single native histogram. #18058
fix[BUGFIX] Agent: fix crash shortly after startup from invalid type of object. #17802
fix[BUGFIX] Scraping: fix relabel keep/drop not working. #17807
other[CHANGE] Native Histograms are no longer experimental! Make the native-histogram feature flag a no-op. Use scrape_native_histograms config option instead. #17528
feature[CHANGE] API: Add maximum limit of 10,000 sets of statistics to TSDB status endpoint. #17647
feature[FEATURE] API: Add /api/v1/features for clients to understand which features are supported. #17427
feature[FEATURE] Promtool: Add start_timestamp field for unit tests. #17636
feature[FEATURE] Promtool: Add --format seriesjson option to tsdb dump to output just series labels in JSON format. #13409
feature[FEATURE] Add --storage.tsdb.delay-compact-file.path flag for better interoperability with Thanos. #17435
feature[FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. #17714
feature[ENHANCEMENT]: TSDB: add flag --storage.tsdb.block-reload-interval to configure TSDB Block Reload Interval. #16728
feature[ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. #17565
other[ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. #16834
feature[ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. #17374
feature[ENHANCEMENT] Notifications: Add a histogram prometheus_notifications_latency_histogram_seconds to complement the existing summary. #16637
feature[ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. #17483
feature[ENHANCEMENT] SD: add a config label with job name for most prometheus_sd_refresh metrics. #17138
feature[ENHANCEMENT] TSDB: New histogram prometheus_tsdb_sample_ooo_delta, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. #17477
other[ENHANCEMENT] Remote-read: Validate histograms received via remote-read. #17561
other[PERF] TSDB: Small optimizations to postings index. #17439
other[PERF] Scraping: Speed up relabelling of series. #17530
other[PERF] PromQL: Small optimisations in binary operators. #17524, #17519.
fix[BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in _total. #17682
fix[BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. #17602
fix[BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. #17605
fix[BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. #17678
fix[BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty ignoring() and non-empty grouping. #17643
fix[BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. #17479
fix[BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. #17668
fix[BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. #17708
fix[BUGFIX] PromQL: Fix collision in label_join and label_replace with non-overlapping series. #17703
fix[BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. #17161
fix[BUGFIX] PromQL: Ensure that rate/increase/delta of histograms results in a gauge histogram. #17608
fix[BUGFIX] PromQL: Do not panic while iterating over invalid histograms. #17559
fix[BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. #17533
fix[BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. #17561
fix[BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. #17546
fix[BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. #17472
fix[BUGFIX] remote: Fix Remote Write receiver, so it does not send wrong response headers for v1 flow and cause Prometheus senders to emit false partial error log and metrics. #17683
otherUpgrade to v3.8 and keep the feature flag. Everything should work as before.
otherAt your own pace, set scrape_native_histograms to true in all relevant scrape configs. (There is a global and a per-scrape-config version of scrape_native_histograms, allowing granular control if needed. It is a good idea to also set scrape_native_histograms explicitly to false where you do not…
breakingRemove the feature flag and make sure that everything still works as intended.
otherNow you are ready for an upgrade to the next release (v3.9).
other[CHANGE] Remote-write 2 (receiving): Update to 2.0-rc.4 spec. "created timestamp" (CT) is now called "start timestamp" (ST). #17411
other[CHANGE] TSDB: Native Histogram Custom Bounds with a NaN threshold are now rejected. #17287
other[FEATURE] OAuth2: support jwt-bearer grant-type (RFC7523 3.1). #17592
feature[FEATURE] Dockerfile: Add OpenContainers spec labels to Dockerfile. #16483
feature[FEATURE] SD: Add unified AWS service discovery for ec2, lightsail and ecs services. #17046
other[FEATURE] Native histograms are now a stable, but optional feature, use the scrape_native_histograms config setting. #17232 #17315
other[FEATURE] UI: Support anchored and smoothed keyword in promql editor. #17239
other[FEATURE] UI: Show detailed relabeling steps for each discovered target. #17337
feature[FEATURE] Alerting: Add urlQueryEscape to template functions. #17403
feature[FEATURE] Promtool: Add Remote-Write 2.0 support to promtool push metrics via the --protobuf_message flag. #17417
other[ENHANCEMENT] Clarify the docs about handling negative native histograms. #17249
feature[ENHANCEMENT] Mixin: Add static UID to the remote-write dashboard. #17256
feature[ENHANCEMENT] PromQL: Reconcile mismatched NHCB bounds in Add and Sub. #17278
feature[ENHANCEMENT] Alerting: Add "unknown" state for alerting rules that haven't been evaluated yet. #17282
feature[ENHANCEMENT] Scrape: Allow simultaneous use of classic histogram → NHCB conversion and zero-timestamp ingestion. #17305
feature[ENHANCEMENT] UI: Add smoothed/anchored in explain. #17334
other[ENHANCEMENT] OTLP: De-duplicate any target_info samples with the same timestamp for the same series. #17400
other[ENHANCEMENT] Document use_fips_sts_endpoint in sigv4 config sections. #17304
other[ENHANCEMENT] Document Prometheus Agent. #14519
other[PERF] PromQL: Speed up parsing of variadic functions. #17316
other[PERF] UI: Speed up alerts/rules/... pages by not rendering collapsed content. #17485
other[PERF] UI: Performance improvement when getting label name and values in promql editor. #17194
other[PERF] UI: Speed up /alerts for many firing alerts via virtual scrolling. #17254
fix[BUGFIX] PromQL: Fix slice indexing bug in info function on churning series. #17199
fix[BUGFIX] API: Reduce lock contention on /api/v1/targets. #17306
fix[BUGFIX] PromQL: Consistent handling of gauge vs. counter histograms in aggregations. #17312
fix[BUGFIX] TSDB: Allow NHCB with -Inf as the first custom value. #17320
fix[BUGFIX] UI: Fix duplicate loading of data from the API speed up rendering of some pages. #17357
fix[BUGFIX] Old UI: Fix createExpressionLink to correctly build /graph URLs so links from Alerts/Rules work again. #17365
fix[BUGFIX] PromQL: Avoid panic when parsing malformed info call. #17379
fix[BUGFIX] PromQL: Include histograms when enforcing sample_limit. #17390
fix[BUGFIX] Config: Fix panic if TLS CA file is absent. #17418
fix[BUGFIX] PromQL: Fix histogram_fraction for classic histograms and NHCB if lower bound is in the first bucket. #17424
breaking[BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389
fix[BUGFIX] Fix federation of some native histograms. #17299 #17409
fix[BUGFIX] promtool: check config would fail when --lint=none flag was set. #17399 #17414
fix[BUGFIX] Remote-write: fix a deadlock in the queue resharding logic that can lead to suboptimal queue behavior. #17412
fix[BUGFIX] AWS SD: Fix AWS SDK v2 credentials handling for EC2 and Lightsail discovery. #17355
fix[BUGFIX] AWS SD: Load AWS region from IMDS when not set. #17376
fix[BUGFIX] Relabeling: Fix labelmap action validation with the legacy metric name validation scheme. #17372
fix[BUGFIX] PromQL: Fix parsing failure when anchored and smoothed are used as metric names and label names. #17353
fix[BUGFIX] PromQL: Fix formatting of range vector selectors with smoothed/anchored modifier. #17354
breaking[BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344
deprecated[CHANGE] Remote-write: the following metrics are deprecated:
otherprometheus_remote_storage_samples_in_total, use prometheus_wal_watcher_records_read_total{type="samples"} and prometheus_remote_storage_samples_dropped_total instead
otherprometheus_remote_storage_histograms_in_total, use prometheus_wal_watcher_records_read_total{type=~".*histogram_samples"} and prometheus_remote_storage_histograms_dropped_total instead
otherprometheus_remote_storage_exemplars_in_total, use prometheus_wal_watcher_records_read_total{type="exemplars"} and prometheus_remote_storage_exemplars_dropped_total instead
otherprometheus_remote_storage_highest_timestamp_in_seconds, use the more accurate prometheus_remote_storage_queue_highest_timestamp_seconds instead in dashboards and alerts to properly account for relabeling and for more accuracy. #17065
feature[FEATURE] PromQL: Add support for experimental anchored and smoothed rate behind feature flag promql-extended-range-selectors. #16457
feature[FEATURE] Federation: Add support for native histograms with custom buckets (NHCB). #17215
feature[FEATURE] PromQL: Add first_over_time(...) and ts_of_first_over_time(...) behind feature flag experimental-promql-functions. #16963 #17021
feature[FEATURE] Remote-write: Add support for Azure Workload Identity as an authentication method for the receiver. #16788
feature[FEATURE] Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the type-and-unit-labels feature flag is enabled. #17033
other[FEATURE] OTLP: Write start time of metrics as created time zero samples into TSDB when created-timestamp-zero-ingestion feature flag is enabled. #16951
feature[ENHANCEMENT] PromQL: Add warn-level annotations for counter reset conflicts in certain histogram operations. #17051 #17094
feature[ENHANCEMENT] UI: Add scrape interval and scrape timeout to targets page. #17158
other[ENHANCEMENT] TSDB: Reduce the resolution of native histograms read from chunks or remote read if the schema is exponential. #17213
feature[ENHANCEMENT] Remote write: Add logging for unexpected metadata in sample batches, when metadata entries are found in samples-only batches. #17034 #17082
other[ENHANCEMENT] Rules: Support concurrent evaluation for rules querying ALERTS and ALERTS_FOR_STATE. #17064
feature[ENHANCEMENT] TSDB: Add logs to improve visibility into internal operations. #17074
other[PERF] OTLP: Write directly to TSDB instead of passing through a Remote-Write adapter when receiving OTLP metrics. #16951
other[PERF] OTLP: Reduce number of logs emitted from OTLP endpoint. No need to log duplicate sample errors. #17201
other[PERF] PromQL: Move more work to preprocessing step. #16896
other[PERF] PromQL: Reduce allocations when walking the syntax tree. #16593
other[PERF] TSDB: Optimize appender creation, slightly speeding up startup. #16922
other[PERF] TSDB: Improve speed of querying a series with multiple matchers. #13971
fix[BUGFIX] Alerting: Mutating alerts relabeling (using replace actions, etc.) within a alertmanager_config.alert_relabel_configs block is now scoped correctly and no longer yields altered alerts to subsequent blocks. #17063
fix[BUGFIX] Config: Infer valid escaping scheme when scrape config validation scheme is set. #16923
fix[BUGFIX] TSDB: Correctly handle appending mixed-typed samples to the same series. #17071 #17241 #17290 #17295 #17296
fix[BUGFIX] Remote-write: Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0, log warning. #17146
fix[BUGFIX] TSDB: Fix metadata entries handling on metadata-wal-records experimental feature for native histograms with custom buckets (NHCB) in protobuf scraping. #17156
fix[BUGFIX] TSDB: Ignore Native Histograms with invalid schemas during WAL/WBL replay. #17214
fix[BUGFIX] PromQL: Avoid empty metric names in annotations for histogram_quantile(). #16794
fix[BUGFIX] PromQL: Correct inaccurate character positions in errors for some aggregate expressions. #16996 #17031
fix[BUGFIX] PromQL: Fix info() function on churning series. #17135
fix[BUGFIX] PromQL: Set native histogram to gauge type when subtracting or multiplying/dividing with negative factors. #17004
fix[BUGFIX] TSDB: Reject unsupported native histogram schemas when attempting to append to TSDB. For scrape and remote-write implement reducing the resolution to fit the maximum if the schema is within the -9 to 52. #17189
fix[BUGFIX] Remote-write: Fix HTTP handler to return after writing error response for invalid compression. #17050
fix[BUGFIX] Remote-write: Return HTTP error 400 instead of 5xx for wrongly formatted Native Histograms. #17210
fix[BUGFIX] Scrape: Prevent staleness markers from generating unnecessary series. #16429
fix[BUGFIX] TSDB: Avoid misleading Failed to calculate size of \"wal\" dir error logs during WAL clean-up. #17006
fix[BUGFIX] TSDB: Prevent erroneously dropping series records during WAL checkpoints. #17029
fix[BUGFIX] UI: Fix redirect to path of -web.external-url if -web.route-prefix is configured. #17240
+1 more item in the release notesfeature[FEATURE] PromQL: Add step(), and min() and max() on durations, behind feature flag promql-duration-expr. #16777
feature[FEATURE] API: Add a /v1/status/tsdb/blocks endpoint exposing metadata about loaded blocks. #16695
feature[FEATURE] Templates: Add toDuration() and now() functions. #16619
feature[ENHANCEMENT] Discovery: Add support for attaching namespace metadata to targets. #16831
feature[ENHANCEMENT] OTLP: Support new UnderscoreEscapingWithoutSuffixes strategy via otlp.translation_strategy. #16849
other[ENHANCEMENT] OTLP: Support including scope metadata as metric labels via otlp.promote_scope_metadata. #16878
feature[ENHANCEMENT] OTLP: Add type and unit labels when feature flag type-and-unit-labels is enabled. #16630
other[ENHANCEMENT] Tracing: Send the traceparent HTTP header during scrapes. #16425
feature[ENHANCEMENT] UI: Add option to disable info and warning query messages under Query page settings. #16901
other[ENHANCEMENT] UI: Improve metadata handling for _count/_sum/_bucket suffixes. #16910
other[ENHANCEMENT] TSDB: Track stale series in the Head block via the prometheus_tsdb_head_stale_series metric. #16925
other[PERF] PromQL: Improve performance due to internal optimizations. #16797
fix[BUGFIX] Config: Fix "unknown global name escaping method" error messages produced during config validation. #16801
fix[BUGFIX] Discovery: Fix race condition during shutdown. #16820
fix[BUGFIX] OTLP: Generate target_info samples between the earliest and latest samples per resource. #16737
fix[BUGFIX] PromQL: Fail when NaN is passed as parameter to topk(), bottomk(), limitk() and limit_ratio(). #16725
fix[BUGFIX] PromQL: Fix extrapolation for native counter histograms. #16828
fix[BUGFIX] PromQL: Reduce numerical errors by disabling some optimizations. #16895
fix[BUGFIX] PromQL: Fix inconsistencies when using native histograms in subqueries. #16879
fix[BUGFIX] PromQL: Fix inconsistent annotations for rate() and increase() on histograms when feature flag type-and-unit-labels is enabled. #16915
fix[BUGFIX] Scraping: Fix memory corruption in slicelabels builds. #16946
fix[BUGFIX] TSDB: Fix panic on append when feature flag created-timestamp-zero-ingestion is enabled. #16332
fix[BUGFIX] TSDB: Fix panic on append for native histograms with empty buckets. #16893
security[SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060
security[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650
security[SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934
security[SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935
other[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18792
securityShadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability.
securityBrett Gervasoni for the AzureAD OAuth client_secret vulnerability.
security@iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability.
security[SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18587
security[SECURITY] Remote-Write: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. #18591
security[SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18585
security[SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18589
security[SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18507
other[PERF] Regex: Stop calling Simplify. #17908
otherDocker library updated from 28.2.2 to 28.5.2. #17821
otherBuilt with Go 1.24.11.
feature[FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag type-and-unit-labels. #16228 #16632 #16718 #16743
feature[FEATURE] PromQL: Add ts_of_(min|max|last)_over_time, behind feature flag experimental-promql-functions. #16722 #16733
feature[FEATURE] Scraping: Add global option always_scrape_classic_histograms to scrape a classic histogram even if it is also exposed as native. #16452
feature[FEATURE] OTLP: New config options promote_all_resource_attributes and ignore_resource_attributes. #16426
feature[FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
feature[ENHANCEMENT] Hetzner SD: Add label_selector to filter servers. #16512
other[ENHANCEMENT] PromQL: support non-constant parameter in aggregations like quantile and topk. #16404
other[ENHANCEMENT] UI: Better total target count display when using keep_dropped_targets option. #16604
feature[ENHANCEMENT] UI: Add simple filtering on the /rules page. #16605
other[ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
other[ENHANCEMENT] UI: Clear search field on /targets page. #16567
other[ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
other[ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
other[PERF] PromQL: Reuse histogramStatsIterator where possible. #16686
other[PERF] PromQL: Reuse storage for custom bucket values for native histograms. #16565
other[PERF] UI: Optimize memoization and search debouncing on /targets page. #16589
fix[PERF] UI: Fix full-page re-rendering when opening status nav menu. #16590
other[PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #16365
other[PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
other[PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #16158
other[PERF] Labels: Simpler/faster stringlabels encoding. #16069
other[PERF] Scraping: Reload scrape pools concurrently. #16595 #16783
fix[BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
fix[BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
fix[BUGFIX] PromQL: Fix native histogram last_over_time. #16744
fix[BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
fix[BUGFIX] PromQL: Don't emit a value from histogram_fraction or histogram_quantile if classic and native histograms are present at the same timestamp. #16552
fix[BUGFIX] PromQL: Incorrect rounding of [1001ms] to [1s] and similar. #16478
fix[BUGFIX] PromQL: Fix inconsistent / sometimes negative histogram_count and histogram_sum. #16682
fix[BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
fix[BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
fix[BUGFIX] PromQL: Improve consistency of avg aggregation and avg_over_time. #16569 #16773
fix[BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
fix[BUGFIX] API: Add HTTP Vary: Origin header to responses to avoid cache poisoning. #16008
fix[BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
fix[BUGFIX] Remote-write: For Azure AD auth, allow empty client_id to suppport system assigned managed identity. #16421
fix[BUGFIX] Scraping: Fix rare memory corruption bug. #16623
fix[BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
fix[BUGFIX] OTLP: Default config not respected when otlp: block is unset. #16693
fix[BUGFIX] OTLP receiver: Fix default configuration not being respected if the otlp: block is unset in the config file. #16693
fix[BUGFIX] Parser: Add reproducer for a dangling-reference issue in parsers. #16633
other[CHANGE] Config: Make setting out-of-order native histograms feature (--enable-feature=ooo-native-histograms) a no-op. Out-of-order native histograms are now always enabled when out_of_order_time_window is greater than zero and --enable-feature=native-histograms is set. #16207
feature[FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #15850
feature[FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #16441
feature[FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #16249
feature[FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #16360
other[ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #16095
feature[ENHANCEMENT] TSDB: add prometheus_tsdb_wal_replay_unknown_refs_total and prometheus_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during WAL/WBL replay. #16166
feature[ENHANCEMENT] Scraping: Add config option for escaping scheme request. #16066
feature[ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #16226
other[ENHANCEMENT] Alerting: make batch size configurable (--alertmanager.notification-batch-size). #16254
other[PERF] Kubernetes SD: make endpointSlice discovery more efficient. #16433
fix[BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #16340
fix[BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #16218
fix[BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #16231
fix[BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #16242
fix[BUGFIX] Promtool: Add feature flags for PromQL features. #16443
fix[BUGFIX] Rules: correct logging of alert name & template data. #15093
fix[BUGFIX] PromQL: Use arithmetic mean for histogram_stddev() and histogram_stdvar() . #16444
fix[BUGFIX] Azure SD: Fix panic on malformed log message. #16434 #16210
fix[BUGFIX] Config: Update GOGC before loading TSDB. #16491
feature[FEATURE] PromQL: Implement idelta() and irate() for native histograms. #15853
feature[ENHANCEMENT] Scaleway SD: Add meta_scaleway_instance_public_ipv4_addresses and meta_scaleway_instance_public_ipv6_addresses labels. #14228
other[ENHANCEMENT] TSDB: Reduce locking while reloading blocks. #12920
feature[ENHANCEMENT] PromQL: Allow UTF-8 labels in label_replace(). #15974
other[ENHANCEMENT] Promtool: tsdb create-blocks-from openmetrics can now read from a Pipe. #16011
feature[ENHANCEMENT] Rules: Add support for anchors and aliases in rule files. #14957
other[ENHANCEMENT] Dockerfile: Make /prometheus writable. #16073
other[ENHANCEMENT] API: Include scrape pool name for dropped targets in /api/v1/targets. #16085
other[ENHANCEMENT] UI: Improve time formatting and copying of selectors. #15999 #16165
other[ENHANCEMENT] UI: Bring back vertical grid lines and graph legend series toggling instructions. #16163 #16164
other[ENHANCEMENT] Mixin: The cluster label can be customized using clusterLabel. #15826
other[PERF] TSDB: Optimize some operations on head chunks by taking shortcuts. #12659
other[PERF] TSDB & Agent: Reduce memory footprint during WL replay. #15778
other[PERF] Remote-Write: Reduce memory footprint during WAL replay. #16197
other[PERF] API: Reduce memory footprint during header parsing. #16001
other[PERF] Rules: Improve dependency evaluation, enabling better concurrency. #16039
other[PERF] Scraping: Improve scraping performance for native histograms. #15731
other[PERF] Scraping: Improve parsing of created timestamps. #16072
fix[BUGFIX] Scraping: Bump cache iteration after error to avoid false duplicate detections. #16174
fix[BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #16218
fix[BUGFIX] PromQL: Fix counter reset detection for native histograms. #15902 #15987
fix[BUGFIX] PromQL: Fix inconsistent behavior with an empty range. #15970
fix[BUGFIX] PromQL: Fix inconsistent annotation in quantile_over_time(). #16018
fix[BUGFIX] PromQL: Prevent label_join() from producing duplicates. #15975
fix[BUGFIX] PromQL: Ignore native histograms in scalar(), sort() and sort_desc(). #15964
breaking[BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895
fix[BUGFIX] Alerting: Consider alert relabeling when deciding whether alerts are dropped. #15979
fix[BUGFIX] Config: Set GoGC to the default value in case of an empty configuration. #16052
breaking[BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060
fix[BUGFIX] Scaleway SD: The public IP will no longer be set to __meta_meta_scaleway_instance_public_ipv4 if it is an IPv6 address. #14228
fix[BUGFIX] UI: Display the correct value of Alerting rules' keep_firing_for. #16211