Prometheus 2.43.1 → 3.15.1
72 releases · 29 breaking changes · 37 security fixes · 1 advisory fixed
37security29breaking10deprecated244feature403fix
Vulnerabilities fixed by this upgrade
- 7.5 CVE-2026-42154 Remote read endpoint allows denial of service via crafted snappy payload
Breaking changes
- breaking [BUGFIX] Discovery: delete the stale prometheus_sd_last_update_timestamp_seconds series for a config that is removed on reload. #19131 in 3.15.0
- breaking [BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326 in 3.15.0
- breaking [BUGFIX] Rules: Clean up stale rule_group_last_rule_duration_sum_seconds and rule_group_last_restore_duration_seconds series when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing /metrics cardinality over time. #19107 in 3.14.0
- breaking [BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. #18849 in 3.13.0
- breaking [BUGFIX] prometheus_sd_refresh* and prometheus_sd_discovered_targets metrics for specific scrape jobs are deleted when the scrape job is removed. #17614 in 3.12.0
- breaking [ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. #17598 in 3.11.0
- breaking [BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. #18200 in 3.11.0
- breaking Remove the feature flag and make sure that everything still works as intended. in 3.8.0
- breaking [BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389 in 3.7.3
- breaking [BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344 in 3.7.1
- breaking [BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895 in 3.3.0
- breaking [BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060 in 3.3.0
- breaking [PERF] Remote-Write: Remove interning hook. #15456 in 3.1.0
- breaking [CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed. #15376 in 3.0.0
- breaking [CHANGE] Set the GOMEMLIMIT variable automatically to match the Linux container memory limit. Use --no-auto-gomemlimit to disable it. The auto-gomemlimit feature flag was removed. #15373 in 3.0.0
- breaking [CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a fallback_scrape_protocol in the scrape config. #15136 in 3.0.0
- breaking [CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657 in 3.0.0
- breaking [CHANGE] Kubernetes SD: Remove support for discovery.k8s.io/v1beta1 API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365 in 3.0.0
- breaking [CHANGE] Kubernetes SD: Remove support for networking.k8s.io/v1beta1 API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365 in 3.0.0
- breaking [CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed. #14705 in 3.0.0
- breaking [CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807 in 3.0.0
- breaking [CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed.… in 3.0.0
- breaking [CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead. #14747 in 3.0.0
- breaking [ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers. #13909 in 3.0.0
- breaking tsdb: remove unused option in 2.50.0
- breaking remove obsolete build tag in 2.50.0
- breaking Remove csmarchbanks from remote write owners in 2.50.0
- breaking head.go: Remove an unneeded snapshot trigger that was moved in https:… in 2.49.0
- breaking [BUGFIX] TSDB: Remove double memory snapshot on shutdown. #13110 in 2.45.3
Security changes
- security [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys. in 3.15.1
- security [SECURITY] Build with Go 1.27.3 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template. in 3.15.1
- security [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617). in 3.15.1
- security [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys. in 3.13.5
- security [SECURITY] Build with Go 1.26.10 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template. in 3.13.5
- security [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617). in 3.13.5
- security [SECURITY] Bump OpenTelemetry to v1.45.0 to fix exporter endpoint URLs leaking into info logs (GO-2026-6505). in 3.13.5
- security [SECURITY] Bump google.golang.org/grpc to v1.83.1 to fix HTTP/2 DATA frame fragmentation memory exhaustion (GO-2026-6348). #19834 in 3.13.4
- security [SECURITY] UI: Update vulnerable npm dependencies. #19834 in 3.13.4
- security [SECURITY] Bump github.com/klauspost/compress to v1.18.7 (GO-2026-5841) and golang.org/x/crypto to v0.55.0 (GO-2026-6303). in 3.13.3
- security [SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama in 3.13.2
- security [SECURITY] UI: Bump sanitize-html to fix a cross-site scripting vulnerability (CVE-2026-44990). #18697 in 3.13.0
- security [CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673… in 3.13.0
- security [SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to @hibrian827 for reporting it. #18642 in 3.12.0
- security [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18649 in 3.12.0
- security Shadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability. in 3.11.3
- security Brett Gervasoni for the AzureAD OAuth client_secret vulnerability. in 3.11.3
- security @iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability. in 3.11.3
- security [SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18590 in 3.11.3
- security [SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18584 in 3.11.3
- security [SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18588 in 3.11.3
- security [SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18506 in 3.11.2
- security [ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. #18347 in 3.11.0
- security [SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060 in 3.5.5
- security [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650 in 3.5.4
- security [SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934 in 3.5.4
- security [SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935 in 3.5.4
- security Shadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability. in 3.5.3
- security Brett Gervasoni for the AzureAD OAuth client_secret vulnerability. in 3.5.3
- security @iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability. in 3.5.3
- security [SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18587 in 3.5.3
- security [SECURITY] Remote-Write: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. #18591 in 3.5.3
- security [SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18585 in 3.5.3
- security [SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18589 in 3.5.3
- security [SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18507 in 3.5.2
- security [SECURITY] upgrade golang.org/x/crypto to address reported CVE-2024-45337. #15691 in 3.1.0
- security This release contains security fixes in dependencies and has been built with go1.21.11. #14309. in 2.45.6
Deprecations
- deprecated [CHANGE] Logging: Deprecate --log.level; use runtime.log_level configuration to supply the default level. #19511 in 3.15.0
- deprecated [ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. --enable-feature=xor2-encoding is deprecated; use storage.tsdb.chunk_encoding.floats: xor2 instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461 in 3.15.0
- deprecated [BUGFIX] discovery/kubernetes: Populate __meta_kubernetes_service_loadbalancer_ip from status.loadBalancer.ingress, falling back to deprecated spec.loadBalancerIP. #19404 in 3.15.0
- deprecated [CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 in 3.14.0
- deprecated [CHANGE] Hetzner SD: The meta_hetzner_datacenter label is deprecated for the role robot but kept for backward compatibility, use the meta_hetzner_robot_datacenter label instead. For the role hcloud, the label is deprecated and will stop working after the 1 July 2026. #17850 in 3.11.0
- deprecated [CHANGE] Hetzner SD: The meta_hetzner_hcloud_datacenter_location and meta_hetzner_hcloud_datacenter_location_network_zone labels are deprecated, use the meta_hetzner_hcloud_location and meta_hetzner_hcloud_location_network_zone labels instead. #17850 in 3.11.0
- deprecated [ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. #17829 in 3.11.0
- deprecated [CHANGE] Remote-write: the following metrics are deprecated: in 3.7.0
- deprecated [CHANGE] Remove deprecated remote-write-receiver,promql-at-modifier, and promql-negative-offset feature flags. #13456, #14526 in 3.0.0
- deprecated [CHANGE] Remove deprecated storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention flags. #14640, #14643 in 3.0.0
Every release in between
security[SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
security[SECURITY] Build with Go 1.27.3 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template.
security[SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
fix[CHANGE] PromQL: A range query whose end was not aligned to step caused subqueries inside it to evaluate past the parent's last actual step, inflating peakSamples in the query stats and against the query.max-samples limit, and wasting storage I/O reading samples that were never used in the result.…
deprecated[CHANGE] Logging: Deprecate --log.level; use runtime.log_level configuration to supply the default level. #19511
feature[FEATURE] Configuration: Allow changing the process log level through runtime.log_level on configuration reload. #19511
feature[FEATURE] Prometheus: Add --auto-gomemlimit.refresh-interval flag to periodically re-detect the container or system memory limit and update GOMEMLIMIT at runtime. #18843
feature[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. #12024. #18091
feature[FEATURE] scrape: Implement OM2.0 scrape format. #18606
feature[ENHANCEMENT] Docker SD: Add labels meta_docker_container_image and meta_docker_container_image_id. #19386
feature[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. #19500
deprecated[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. --enable-feature=xor2-encoding is deprecated; use storage.tsdb.chunk_encoding.floats: xor2 instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461
feature[ENHANCEMENT] TSDB: add prometheus_tsdb_head_appenders_created_total metric. #19411
feature[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. #19410
feature[ENHANCEMENT] storage/remote: Add undocumented failed_request_logging config field to debug log remote write V2 requests on send errors. #19249
feature[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. #18049
feature[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. #19332
feature[ENHANCEMENT] TSDB: Add prometheus_tsdb_head_series_pending_commit_underflow_total to report pending-sample reservation underflows. #19470
fix[BUGFIX] PromQL: Fix info() enrichment for composite expressions with mixed @/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. #19387
fix[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. #19557
fix[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. #19413
fix[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. #19272
fix[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. #19470
fix[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. #19470
fix[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. #19664
fix[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. #19460
fix[BUGFIX] TSDB: fix default block reload interval for custom options. #19368
fix[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. #19194
fix[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. #19435
fix[BUGFIX] AWS SD: Reject non-positive request_concurrency instead of hanging service discovery indefinitely. #19524
fix[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. #19302
fix[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. #19324
breaking[BUGFIX] Discovery: delete the stale prometheus_sd_last_update_timestamp_seconds series for a config that is removed on reload. #19131
fix[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. #19661
fix[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. #19438
fix[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. #19218
fix[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. #19444
fix[BUGFIX] Native histograms: DetectReset no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram rate()/increase() undercount. #19367
fix[BUGFIX] Never skip histogram buckets for histogram_stddev and histogram_stdvar functions. #19521
fix[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. #19343
fix[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. .\|(foo)\|.). #19516
breaking[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326
deprecated[BUGFIX] discovery/kubernetes: Populate __meta_kubernetes_service_loadbalancer_ip from status.loadBalancer.ingress, falling back to deprecated spec.loadBalancerIP. #19404
+40 more items in the release notesdeprecated[CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124
other[CHANGE] API: /api/v1/status/config now correctly shows separator: "" and replacement: "" in relabel configs when explicitly set to empty, instead of omitting them. #18653
other[CHANGE] Discovery/Hetzner: Drop the __meta_hetzner_datacenter label for hcloud targets, following its removal from the Hetzner Cloud API. #19269
other[CHANGE] PromQL: Enable duration expressions by default. The promql-duration-expr feature flag is now a no-op. #19033
other[CHANGE] PromQL: Promote first_over_time to stable. It no longer requires the promql-experimental-functions feature flag. #19093
feature[FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (oci_sd_configs). #18919
feature[FEATURE] PromQL: Add experimental start_timestamp(instant-vector) function returning the start timestamp of each sample in the given vector. Requires the use-start-timestamps feature flag. #19089
feature[FEATURE] PromQL: Allow rate() and increase() to use start timestamps as an alternative for rate extrapolation. Hidden behind the use-start-timestamps feature flag. #18619
feature[FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the histograms-st-encoding feature flag. #18609
other[ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g. k8s.pod.name and k8s_pod_name both become k8s_pod_name), and expose the prometheus_api_otlp_translation_warnings_total counter labelled by category to track such warnings.…
feature[ENHANCEMENT] Promtool: Add --remote-write.path flag to push metrics for backends that use a non-default remote-write endpoint. #19086
other[ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903
feature[ENHANCEMENT] TSDB: Add prometheus_tsdb_head_native_histogram_series and prometheus_tsdb_head_native_histogram_buckets gauges tracking the number of native histogram series and buckets in the head. #19170
feature[ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (step(), range(), min_of(), max_of()) in range selectors and subqueries. #18625
feature[ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706
other[ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181
other[PERF] Speed up regex label matchers matching a set of literal values (e.g. {job=~"foo|bar|baz"}). #18833
other[PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470
other[PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143
fix[BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
fix[BUGFIX] Discovery/AWS: Stop promtool check config from making AWS metadata service (IMDS) network calls when the region field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037
fix[BUGFIX] Discovery/Docker: Set a request timeout for docker_sd and dockerswarm_sd on unix, npipe, and tcp hosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237
fix[BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
fix[BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778
fix[BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
fix[BUGFIX] PromQL: Fix mad_over_time returning 0 instead of NaN when the range contains a NaN sample. #19040
fix[BUGFIX] Promtool: Accept --enable-feature=promql-binop-fill-modifiers in check rules, which previously rejected valid fill()/fill_left()/fill_right() expressions. #19153
fix[BUGFIX] Remote write: Respect the AZURE_FEDERATED_TOKEN_FILE environment variable for workload identity authentication instead of hardcoding the token file path. #18973
breaking[BUGFIX] Rules: Clean up stale rule_group_last_rule_duration_sum_seconds and rule_group_last_restore_duration_seconds series when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing /metrics cardinality over time. #19107
fix[BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
fix[BUGFIX] TSDB: Fix silent data loss and potential crash loop when stale_series_compaction_threshold is used in the config file. #19016
fix[BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted. #19016
fix[BUGFIX] TSDB: Fix prometheus_tsdb_head_stale_series over-counting and early eviction of series that change between float, integer histogram, and float histogram sample types. #19183
fix[BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
fix[BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
fix[BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
fix[BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart. #19202
fix[BUGFIX] TSDB: Surface query errors that were previously silently discarded. #19120
fix[BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with --enable-feature=xor2-encoding could silently revert to XOR after compaction. #19145
fix[BUGFIX] UI: Show the delete-series form on the TSDB Status page when --web.enable-admin-api is enabled. #19025
+2 more items in the release notessecurity[SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
security[SECURITY] Build with Go 1.26.10 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template.
security[SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
security[SECURITY] Bump OpenTelemetry to v1.45.0 to fix exporter endpoint URLs leaking into info logs (GO-2026-6505).
other[CHANGE] Building Prometheus now requires Go 1.26 or later.
security[SECURITY] Bump google.golang.org/grpc to v1.83.1 to fix HTTP/2 DATA frame fragmentation memory exhaustion (GO-2026-6348). #19834
security[SECURITY] UI: Update vulnerable npm dependencies. #19834
fix[BUGFIX] Agent: Ignore unknown WAL record types to allow rolling back from newer versions. #19814
fix[BUGFIX] Federation: Fix corruption of float native histograms. #19679
fix[BUGFIX] Scrape: Fix failing scrapes of protobuf float histograms with zero sample, zero and bucket counts, such as the result of recording rate(x[1m]) over a constant histogram. #19682
security[SECURITY] Bump github.com/klauspost/compress to v1.18.7 (GO-2026-5841) and golang.org/x/crypto to v0.55.0 (GO-2026-6303).
fix[BUGFIX] Docker SD: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
fix[BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
fix[BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
fix[BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
fix[BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
fix[BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
fix[BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
security[SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama
fix[BUGFIX] PromQL: Preallocate the active query tracker file to avoid SIGBUS crashes when the data disk is full. #19289 by @akshajrawat
fix[BUGFIX] TSDB: Fix the head-chunk cache returning samples from the wrong chunk, or spurious not-found errors, to range queries after head-chunk truncation. #19134
security[SECURITY] UI: Bump sanitize-html to fix a cross-site scripting vulnerability (CVE-2026-44990). #18697
other[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at /assets/third-party-licenses.txt, replacing the npm_licenses.tar.bz2 archive previously shipped in release tarballs and container images. #18997
other[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. #18927
security[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673…
other[CHANGE] promtool: Relative file paths in the file passed to --http.config.file are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. #18949
other[CHANGE] PromQL: Rename the min() and max() duration-expression functions (experimental feature flag experimental-duration-expr) to min_of() and max_of() to avoid confusion with the min and max aggregate operators. #18687
feature[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. #18573
feature[FEATURE] Discovery/AWS: Add ability to filter RDS instances. #18859
feature[FEATURE] PromQL: Add min_of(a, b) and max_of(a, b) scalar experimental functions, returning the smaller or larger of two scalar values. #18687
feature[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. #18564
feature[FEATURE] PromQL: Expose per-query samplesRead (and samplesReadPerStep with stats=all and the promql-per-step-stats feature flag) in the query stats response, and add the prometheus_engine_query_samples_read_total engine counter. samplesRead reflects storage I/O distinct from…
feature[FEATURE] Scrape: Add convert_classic_histograms_to_nhcb internal label to allow per-target override of convert_classic_histograms_to_nhcb scrape configuration via relabeling. #18840
feature[FEATURE] TSDB: Add storage.tsdb.chunk_encoding.floats configuration field to select float chunk encoding (xor or xor2) at runtime, independently of the --enable-feature=xor2-encoding flag. #18769
feature[FEATURE] remote_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. #18217
feature[FEATURE] Scrape: Add always_scrape_classic_histograms and scrape_native_histograms internal labels to allow per-target override of the always_scrape_classic_histograms and scrape_native_histograms scrape configuration via relabeling. #18929
other[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18791
other[ENHANCEMENT] PromQL: Prettify fill_left(x) fill_right(x) as fill(x) when both fill values are equal. #18851
other[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. #18894
feature[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to ~2x faster). #18540
other[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by ~12-15% in benchmarks. #18699
other[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (--enable-feature=created-timestamp-zero-ingestion). #18813
fix[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. #18845
fix[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. #18629
fix[BUGFIX] PromQL: A range query whose end was not aligned to step caused subqueries inside it to evaluate past the parent's last actual step, inflating peakSamples in the query stats and against the query.max-samples limit, and wasting storage I/O reading samples that were never used in the result.…
fix[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an @ modifier (e.g. predict_linear(metric[60s] @ T, X)) silently under-counted totalQueryableSamples for steps after step 0. #18081
fix[BUGFIX] PromQL: Fix fill_left/fill_right producing missing samples in range queries when using group_left/group_right. #18850
fix[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. #18906
fix[BUGFIX] PromQL: Fix panic on 1[5m] smoothed and similar expressions when extended range selectors are enabled. #18764
fix[BUGFIX] PromQL: Fix panic when a smoothed instant vector selector produces no samples for a series. #18943
fix[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. foo offset -(5)). #18768
fix[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces {}. Via prometheus/common v0.69.0. #18949
fix[BUGFIX] Promtool: Fix check healthy and check ready when --url ends with a trailing slash. #18854
fix[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. #18733
fix[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy private_ip or public_ip field, but do have private NICs attached. #18772
fix[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). #18838
fix[BUGFIX] UI: Escape label values offered by PromQL autocomplete. #18658
fix[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. #18739
fix[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. #18847
breaking[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. #18849
featureSecurity: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery.
featurePromQL & Metadata: Several features and bug fixes related to the experimental "start timestamps" support, including updates to rate(), irate(), increase(), and resets(). New experimental functions start(), end(), range(), and step() are introduced.
featureTSDB Performance: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage.
featureService Discovery: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support).
featureUI: Added a web interface for deleting time series and cleaning tombstones.
security[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to @hibrian827 for reporting it. #18642
security[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18649
feature[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used st-storage flag is enabled. #18221
feature[FEATURE] API: Add /api/v1/status/self_metrics endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. #18411
feature[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery #18287
feature[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target #18321
feature[FEATURE] Discovery: Add Outscale VM service discovery (outscale_sd_configs) for discovering scrape targets from the Outscale Cloud API. #18139
feature[FEATURE] PromQL: Add start(), end(), range(), and step() experimental functions #17877
feature[FEATURE] TSDB/Agent: Add CheckpointFromInMemorySeries option to agent.DB that enables checkpoint based on in-memory series. #17948
feature[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. #18390
feature[FEATURE] Scrape: Added a feature flag st-synthesis which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. #18279
feature[ENHANCEMENT] AWS SD: Add optional external_id field to ECS/MSK/RDS/Elasticache. #18579
feature[ENHANCEMENT] AWS SD: Add optional external_id field. #17171
feature[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. #18187
feature[ENHANCEMENT] Promtool: Add --header flag to query instant command, matching existing query range behaviour. #18418
feature[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. #16088
fix[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. #18568
fix[BUGFIX] PromQL: Fix info() function incorrectly handling negated name matchers #17932
fix[BUGFIX] API: Return duration expressions in /parse_ast. #18624
fix[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec #18305
fix[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 #18567
fix[BUGFIX] Config: Validate remote_write queue_config fields at load time to prevent runtime panic and silent misconfiguration. #18209
fix[BUGFIX] Discovery/Consul: Add health_filter for Health API filtering, fixing breakage when using Catalog-only fields like ServiceTags in filter. #18479 #18499
fix[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. #18408
fix[BUGFIX] PromQL: Fix smoothed rate/increase returning zero instead of no result when all data falls strictly after the query range. #18523
fix[BUGFIX] PromQL: Fix metric name not being dropped when last_over_time or first_over_time is applied to subqueries containing name-dropping functions like abs(). #18409
fix[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. #18660
fix[BUGFIX] PromQL: Fix parsing of range() keyword in duration expressions such as foo[5m+range()]. #18623
fix[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the @ modifier is used. #18531
fix[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. #18639
fix[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. #18414
fix[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. #18382
fix[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. #18421
fix[BUGFIX] TSDB: Allow retention percentage with new data path. #18628
breaking[BUGFIX] prometheus_sd_refresh* and prometheus_sd_discovered_targets metrics for specific scrape jobs are deleted when the scrape job is removed. #17614
+18 more items in the release notessecurityShadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability.
securityBrett Gervasoni for the AzureAD OAuth client_secret vulnerability.
security@iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability.
security[SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18590
security[SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18584
security[SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18588
security[SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18506
feature[ENHANCEMENT] Consul SD: Introduce health_filter field for Health API filtering. #18499
fix[BUGFIX] Consul SD: Fix filter parameter being incorrectly applied to the Health API. #18499
fix[BUGFIX] Tracing: Fix startup failure for OTLP HTTP tracing with insecure: true. #18469
deprecated[CHANGE] Hetzner SD: The meta_hetzner_datacenter label is deprecated for the role robot but kept for backward compatibility, use the meta_hetzner_robot_datacenter label instead. For the role hcloud, the label is deprecated and will stop working after the 1 July 2026. #17850
deprecated[CHANGE] Hetzner SD: The meta_hetzner_hcloud_datacenter_location and meta_hetzner_hcloud_datacenter_location_network_zone labels are deprecated, use the meta_hetzner_hcloud_location and meta_hetzner_hcloud_location_network_zone labels instead. #17850
feature[FEATURE] AWS SD: Add Elasticache Role. #18099
feature[FEATURE] AWS SD: Add RDS Role. #18206
feature[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. #17207
feature[FEATURE] Discovery: Introduce prometheus_sd_last_update_timestamp_seconds metric to track the last time a service discovery update was sent to consumers. #18194
feature[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. #18006
feature[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: meta_kubernetes_pod_deployment_name, meta_kubernetes_pod_cronjob_name and __meta_kubernetes_pod_job_name, respectively. #17774
feature[FEATURE] PromQL: Add / operators for trimming observations from native histograms. #17904
feature[FEATURE] PromQL: Add experimental histogram_quantiles variadic function for computing multiple quantiles at once. #17285
feature[FEATURE] TSDB: Add storage.tsdb.retention.percentage configuration to configure the maximum percent of disk usable for TSDB storage. #18080
feature[FEATURE] TSDB: Add an experimental st-storage feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. #18062
feature[FEATURE] TSDB: Add an experimental xor2-encoding feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. #18062
feature[ENHANCEMENT] HTTP client: Add AWS external_id support for sigv4. #17916
deprecated[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. #17829
breaking[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. #17598
feature[ENHANCEMENT] UI: Add autocomplete support for experimental first_over_time and ts_of_first_over_time PromQL functions. #18318
security[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. #18347
fix[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using KahanAdd. #18252
fix[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured endpoint option, a regression from the AWS SDK v2 migration. #18133
fix[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. #18133
fix[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. #17538
fix[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. #18244
fix[BUGFIX] Azure SD: Fix system-assigned managed identity not working when client_id is empty. #18323
fix[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. #17349
fix[BUGFIX] Kubernetes SD: Fix duplicate targets generated by *DualStack EndpointSlices policies. #18192
fix[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. #18084
fix[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. #18056
fix[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. #18263
fix[BUGFIX] Remote write: Fix prometheus_remote_storage_sent_batch_duration_seconds measuring before the request was sent. #18214
fix[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. #18375
fix[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. #18229
fix[BUGFIX] TSDB: Fail early when use-uncached-io feature flag is set on unsupported environments. #18219
breaking[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. #18200
fix[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. #17895
fix[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. #18306
fix[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. #18200
fix[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. #18189
fix[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. #18228
fix[BUGFIX] UI: Update retention display in runtime info when config is reloaded. #18200
+5 more items in the release notesfeature[CHANGE] Alerting: Add alertmanager dimension to following metrics: prometheus_notifications_dropped_total, prometheus_notifications_queue_capacity, prometheus_notifications_queue_length. #16355
other[CHANGE] UI: Hide expanded alert annotations by default, enabling more information density on the /alerts page. #17611
feature[FEATURE] AWS SD: Add MSK Role. #17600
feature[FEATURE] PromQL: Add fill() / fill_left() / fill_right() binop modifiers for specifying default values for missing series. #17644
feature[FEATURE] Web: Add OpenAPI 3.2 specification for the HTTP API at /api/v1/openapi.yaml. #17825
feature[FEATURE] Dockerfile: Add distroless image variant using UID/GID 65532 and no VOLUME declaration. Busybox image remains default. #17876
feature[FEATURE] Web: Add on-demand wall time profiling under /debug/pprof/fgprof. #18027
feature[ENHANCEMENT] PromQL: Add more detail to histogram quantile monotonicity info annotations. #15578
other[ENHANCEMENT] Alerting: Independent alertmanager sendloops. #16355
feature[ENHANCEMENT] TSDB: Experimental support for early compaction of stale series in the memory with configurable threshold stale_series_compaction_threshold in the config file. #16929
feature[ENHANCEMENT] Service Discovery: Service discoveries are now removable from the Prometheus binary through the Go build tag remove_all_sd and individual service discoveries can be re-added with the build tags enable__sd. Users can build a custom Prometheus with only the necessary SDs for a smaller…
other[ENHANCEMENT] Promtool: Support promql syntax features promql-duration-expr and promql-extended-range-selectors. #17926
other[PERF] PromQL: Avoid unnecessary label extraction in PromQL functions. #17676
other[PERF] PromQL: Improve performance of regex matchers like .-.-.*. #17707
feature[PERF] OTLP: Add label caching for OTLP-to-Prometheus conversion to reduce allocations and improve latency. #17860
other[PERF] API: Compute /api/v1/targets/relabel_steps in a single pass instead of re-running relabeling for each prefix. #17969
other[PERF] tsdb: Optimize LabelValues intersection performance for matchers. #18069
fix[BUGFIX] PromQL: Prevent query strings containing only UTF-8 continuation bytes from crashing Prometheus. #17735
fix[BUGFIX] Web: Fix missing X-Prometheus-Stopping header for /-/ready endpoint in NotReady state. #17795
fix[BUGFIX] PromQL: Fix PromQL info() function returning empty results when filtering by a label that exists on both the input metric and target_info. #17817
fix[BUGFIX] TSDB: Fix a bug during exemplar buffer grow/shrink that could cause exemplars to be incorrectly discarded. #17863
fix[BUGFIX] UI: Fix broken graph display after page reload, due to broken Y axis min encoding/decoding. #17869
fix[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields (Labels, Histogram pointers, metadata strings) before returning buffers to pools. #17879
fix[BUGFIX] PromQL: info function: fix series without identifying labels not being returned. #17898
fix[BUGFIX] OTLP: Filter name from OTLP attributes to prevent duplicate labels. #17917
fix[BUGFIX] TSDB: Fix division by zero when computing stale series ratio with empty head. #17952
fix[BUGFIX] OTLP: Fix potential silent data loss for sum metrics. #17954
fix[BUGFIX] PromQL: Fix smoothed interpolation across counter resets. #17988
fix[BUGFIX] PromQL: Fix panic with @ modifier on empty ranges. #18020
fix[BUGFIX] PromQL: Fix avg_over_time for a single native histogram. #18058
fix[BUGFIX] Agent: fix crash shortly after startup from invalid type of object. #17802
fix[BUGFIX] Scraping: fix relabel keep/drop not working. #17807
other[CHANGE] Native Histograms are no longer experimental! Make the native-histogram feature flag a no-op. Use scrape_native_histograms config option instead. #17528
feature[CHANGE] API: Add maximum limit of 10,000 sets of statistics to TSDB status endpoint. #17647
feature[FEATURE] API: Add /api/v1/features for clients to understand which features are supported. #17427
feature[FEATURE] Promtool: Add start_timestamp field for unit tests. #17636
feature[FEATURE] Promtool: Add --format seriesjson option to tsdb dump to output just series labels in JSON format. #13409
feature[FEATURE] Add --storage.tsdb.delay-compact-file.path flag for better interoperability with Thanos. #17435
feature[FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. #17714
feature[ENHANCEMENT]: TSDB: add flag --storage.tsdb.block-reload-interval to configure TSDB Block Reload Interval. #16728
feature[ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. #17565
other[ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. #16834
feature[ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. #17374
feature[ENHANCEMENT] Notifications: Add a histogram prometheus_notifications_latency_histogram_seconds to complement the existing summary. #16637
feature[ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. #17483
feature[ENHANCEMENT] SD: add a config label with job name for most prometheus_sd_refresh metrics. #17138
feature[ENHANCEMENT] TSDB: New histogram prometheus_tsdb_sample_ooo_delta, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. #17477
other[ENHANCEMENT] Remote-read: Validate histograms received via remote-read. #17561
other[PERF] TSDB: Small optimizations to postings index. #17439
other[PERF] Scraping: Speed up relabelling of series. #17530
other[PERF] PromQL: Small optimisations in binary operators. #17524, #17519.
fix[BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in _total. #17682
fix[BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. #17602
fix[BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. #17605
fix[BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. #17678
fix[BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty ignoring() and non-empty grouping. #17643
fix[BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. #17479
fix[BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. #17668
fix[BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. #17708
fix[BUGFIX] PromQL: Fix collision in label_join and label_replace with non-overlapping series. #17703
fix[BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. #17161
fix[BUGFIX] PromQL: Ensure that rate/increase/delta of histograms results in a gauge histogram. #17608
fix[BUGFIX] PromQL: Do not panic while iterating over invalid histograms. #17559
fix[BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. #17533
fix[BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. #17561
fix[BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. #17546
fix[BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. #17472
fix[BUGFIX] remote: Fix Remote Write receiver, so it does not send wrong response headers for v1 flow and cause Prometheus senders to emit false partial error log and metrics. #17683
otherUpgrade to v3.8 and keep the feature flag. Everything should work as before.
otherAt your own pace, set scrape_native_histograms to true in all relevant scrape configs. (There is a global and a per-scrape-config version of scrape_native_histograms, allowing granular control if needed. It is a good idea to also set scrape_native_histograms explicitly to false where you do not…
breakingRemove the feature flag and make sure that everything still works as intended.
otherNow you are ready for an upgrade to the next release (v3.9).
other[CHANGE] Remote-write 2 (receiving): Update to 2.0-rc.4 spec. "created timestamp" (CT) is now called "start timestamp" (ST). #17411
other[CHANGE] TSDB: Native Histogram Custom Bounds with a NaN threshold are now rejected. #17287
other[FEATURE] OAuth2: support jwt-bearer grant-type (RFC7523 3.1). #17592
feature[FEATURE] Dockerfile: Add OpenContainers spec labels to Dockerfile. #16483
feature[FEATURE] SD: Add unified AWS service discovery for ec2, lightsail and ecs services. #17046
other[FEATURE] Native histograms are now a stable, but optional feature, use the scrape_native_histograms config setting. #17232 #17315
other[FEATURE] UI: Support anchored and smoothed keyword in promql editor. #17239
other[FEATURE] UI: Show detailed relabeling steps for each discovered target. #17337
feature[FEATURE] Alerting: Add urlQueryEscape to template functions. #17403
feature[FEATURE] Promtool: Add Remote-Write 2.0 support to promtool push metrics via the --protobuf_message flag. #17417
other[ENHANCEMENT] Clarify the docs about handling negative native histograms. #17249
feature[ENHANCEMENT] Mixin: Add static UID to the remote-write dashboard. #17256
feature[ENHANCEMENT] PromQL: Reconcile mismatched NHCB bounds in Add and Sub. #17278
feature[ENHANCEMENT] Alerting: Add "unknown" state for alerting rules that haven't been evaluated yet. #17282
feature[ENHANCEMENT] Scrape: Allow simultaneous use of classic histogram → NHCB conversion and zero-timestamp ingestion. #17305
feature[ENHANCEMENT] UI: Add smoothed/anchored in explain. #17334
other[ENHANCEMENT] OTLP: De-duplicate any target_info samples with the same timestamp for the same series. #17400
other[ENHANCEMENT] Document use_fips_sts_endpoint in sigv4 config sections. #17304
other[ENHANCEMENT] Document Prometheus Agent. #14519
other[PERF] PromQL: Speed up parsing of variadic functions. #17316
other[PERF] UI: Speed up alerts/rules/... pages by not rendering collapsed content. #17485
other[PERF] UI: Performance improvement when getting label name and values in promql editor. #17194
other[PERF] UI: Speed up /alerts for many firing alerts via virtual scrolling. #17254
fix[BUGFIX] PromQL: Fix slice indexing bug in info function on churning series. #17199
fix[BUGFIX] API: Reduce lock contention on /api/v1/targets. #17306
fix[BUGFIX] PromQL: Consistent handling of gauge vs. counter histograms in aggregations. #17312
fix[BUGFIX] TSDB: Allow NHCB with -Inf as the first custom value. #17320
fix[BUGFIX] UI: Fix duplicate loading of data from the API speed up rendering of some pages. #17357
fix[BUGFIX] Old UI: Fix createExpressionLink to correctly build /graph URLs so links from Alerts/Rules work again. #17365
fix[BUGFIX] PromQL: Avoid panic when parsing malformed info call. #17379
fix[BUGFIX] PromQL: Include histograms when enforcing sample_limit. #17390
fix[BUGFIX] Config: Fix panic if TLS CA file is absent. #17418
fix[BUGFIX] PromQL: Fix histogram_fraction for classic histograms and NHCB if lower bound is in the first bucket. #17424
breaking[BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389
fix[BUGFIX] Fix federation of some native histograms. #17299 #17409
fix[BUGFIX] promtool: check config would fail when --lint=none flag was set. #17399 #17414
fix[BUGFIX] Remote-write: fix a deadlock in the queue resharding logic that can lead to suboptimal queue behavior. #17412
fix[BUGFIX] AWS SD: Fix AWS SDK v2 credentials handling for EC2 and Lightsail discovery. #17355
fix[BUGFIX] AWS SD: Load AWS region from IMDS when not set. #17376
fix[BUGFIX] Relabeling: Fix labelmap action validation with the legacy metric name validation scheme. #17372
fix[BUGFIX] PromQL: Fix parsing failure when anchored and smoothed are used as metric names and label names. #17353
fix[BUGFIX] PromQL: Fix formatting of range vector selectors with smoothed/anchored modifier. #17354
breaking[BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344
deprecated[CHANGE] Remote-write: the following metrics are deprecated:
otherprometheus_remote_storage_samples_in_total, use prometheus_wal_watcher_records_read_total{type="samples"} and prometheus_remote_storage_samples_dropped_total instead
otherprometheus_remote_storage_histograms_in_total, use prometheus_wal_watcher_records_read_total{type=~".*histogram_samples"} and prometheus_remote_storage_histograms_dropped_total instead
otherprometheus_remote_storage_exemplars_in_total, use prometheus_wal_watcher_records_read_total{type="exemplars"} and prometheus_remote_storage_exemplars_dropped_total instead
otherprometheus_remote_storage_highest_timestamp_in_seconds, use the more accurate prometheus_remote_storage_queue_highest_timestamp_seconds instead in dashboards and alerts to properly account for relabeling and for more accuracy. #17065
feature[FEATURE] PromQL: Add support for experimental anchored and smoothed rate behind feature flag promql-extended-range-selectors. #16457
feature[FEATURE] Federation: Add support for native histograms with custom buckets (NHCB). #17215
feature[FEATURE] PromQL: Add first_over_time(...) and ts_of_first_over_time(...) behind feature flag experimental-promql-functions. #16963 #17021
feature[FEATURE] Remote-write: Add support for Azure Workload Identity as an authentication method for the receiver. #16788
feature[FEATURE] Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the type-and-unit-labels feature flag is enabled. #17033
other[FEATURE] OTLP: Write start time of metrics as created time zero samples into TSDB when created-timestamp-zero-ingestion feature flag is enabled. #16951
feature[ENHANCEMENT] PromQL: Add warn-level annotations for counter reset conflicts in certain histogram operations. #17051 #17094
feature[ENHANCEMENT] UI: Add scrape interval and scrape timeout to targets page. #17158
other[ENHANCEMENT] TSDB: Reduce the resolution of native histograms read from chunks or remote read if the schema is exponential. #17213
feature[ENHANCEMENT] Remote write: Add logging for unexpected metadata in sample batches, when metadata entries are found in samples-only batches. #17034 #17082
other[ENHANCEMENT] Rules: Support concurrent evaluation for rules querying ALERTS and ALERTS_FOR_STATE. #17064
feature[ENHANCEMENT] TSDB: Add logs to improve visibility into internal operations. #17074
other[PERF] OTLP: Write directly to TSDB instead of passing through a Remote-Write adapter when receiving OTLP metrics. #16951
other[PERF] OTLP: Reduce number of logs emitted from OTLP endpoint. No need to log duplicate sample errors. #17201
other[PERF] PromQL: Move more work to preprocessing step. #16896
other[PERF] PromQL: Reduce allocations when walking the syntax tree. #16593
other[PERF] TSDB: Optimize appender creation, slightly speeding up startup. #16922
other[PERF] TSDB: Improve speed of querying a series with multiple matchers. #13971
fix[BUGFIX] Alerting: Mutating alerts relabeling (using replace actions, etc.) within a alertmanager_config.alert_relabel_configs block is now scoped correctly and no longer yields altered alerts to subsequent blocks. #17063
fix[BUGFIX] Config: Infer valid escaping scheme when scrape config validation scheme is set. #16923
fix[BUGFIX] TSDB: Correctly handle appending mixed-typed samples to the same series. #17071 #17241 #17290 #17295 #17296
fix[BUGFIX] Remote-write: Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0, log warning. #17146
fix[BUGFIX] TSDB: Fix metadata entries handling on metadata-wal-records experimental feature for native histograms with custom buckets (NHCB) in protobuf scraping. #17156
fix[BUGFIX] TSDB: Ignore Native Histograms with invalid schemas during WAL/WBL replay. #17214
fix[BUGFIX] PromQL: Avoid empty metric names in annotations for histogram_quantile(). #16794
fix[BUGFIX] PromQL: Correct inaccurate character positions in errors for some aggregate expressions. #16996 #17031
fix[BUGFIX] PromQL: Fix info() function on churning series. #17135
fix[BUGFIX] PromQL: Set native histogram to gauge type when subtracting or multiplying/dividing with negative factors. #17004
fix[BUGFIX] TSDB: Reject unsupported native histogram schemas when attempting to append to TSDB. For scrape and remote-write implement reducing the resolution to fit the maximum if the schema is within the -9 to 52. #17189
fix[BUGFIX] Remote-write: Fix HTTP handler to return after writing error response for invalid compression. #17050
fix[BUGFIX] Remote-write: Return HTTP error 400 instead of 5xx for wrongly formatted Native Histograms. #17210
fix[BUGFIX] Scrape: Prevent staleness markers from generating unnecessary series. #16429
fix[BUGFIX] TSDB: Avoid misleading Failed to calculate size of \"wal\" dir error logs during WAL clean-up. #17006
fix[BUGFIX] TSDB: Prevent erroneously dropping series records during WAL checkpoints. #17029
fix[BUGFIX] UI: Fix redirect to path of -web.external-url if -web.route-prefix is configured. #17240
+1 more item in the release notesfeature[FEATURE] PromQL: Add step(), and min() and max() on durations, behind feature flag promql-duration-expr. #16777
feature[FEATURE] API: Add a /v1/status/tsdb/blocks endpoint exposing metadata about loaded blocks. #16695
feature[FEATURE] Templates: Add toDuration() and now() functions. #16619
feature[ENHANCEMENT] Discovery: Add support for attaching namespace metadata to targets. #16831
feature[ENHANCEMENT] OTLP: Support new UnderscoreEscapingWithoutSuffixes strategy via otlp.translation_strategy. #16849
other[ENHANCEMENT] OTLP: Support including scope metadata as metric labels via otlp.promote_scope_metadata. #16878
feature[ENHANCEMENT] OTLP: Add type and unit labels when feature flag type-and-unit-labels is enabled. #16630
other[ENHANCEMENT] Tracing: Send the traceparent HTTP header during scrapes. #16425
feature[ENHANCEMENT] UI: Add option to disable info and warning query messages under Query page settings. #16901
other[ENHANCEMENT] UI: Improve metadata handling for _count/_sum/_bucket suffixes. #16910
other[ENHANCEMENT] TSDB: Track stale series in the Head block via the prometheus_tsdb_head_stale_series metric. #16925
other[PERF] PromQL: Improve performance due to internal optimizations. #16797
fix[BUGFIX] Config: Fix "unknown global name escaping method" error messages produced during config validation. #16801
fix[BUGFIX] Discovery: Fix race condition during shutdown. #16820
fix[BUGFIX] OTLP: Generate target_info samples between the earliest and latest samples per resource. #16737
fix[BUGFIX] PromQL: Fail when NaN is passed as parameter to topk(), bottomk(), limitk() and limit_ratio(). #16725
fix[BUGFIX] PromQL: Fix extrapolation for native counter histograms. #16828
fix[BUGFIX] PromQL: Reduce numerical errors by disabling some optimizations. #16895
fix[BUGFIX] PromQL: Fix inconsistencies when using native histograms in subqueries. #16879
fix[BUGFIX] PromQL: Fix inconsistent annotations for rate() and increase() on histograms when feature flag type-and-unit-labels is enabled. #16915
fix[BUGFIX] Scraping: Fix memory corruption in slicelabels builds. #16946
fix[BUGFIX] TSDB: Fix panic on append when feature flag created-timestamp-zero-ingestion is enabled. #16332
fix[BUGFIX] TSDB: Fix panic on append for native histograms with empty buckets. #16893
security[SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060
security[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650
security[SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934
security[SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935
other[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18792
securityShadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability.
securityBrett Gervasoni for the AzureAD OAuth client_secret vulnerability.
security@iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability.
security[SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18587
security[SECURITY] Remote-Write: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. #18591
security[SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18585
security[SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18589
security[SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18507
other[PERF] Regex: Stop calling Simplify. #17908
otherDocker library updated from 28.2.2 to 28.5.2. #17821
otherBuilt with Go 1.24.11.
feature[FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag type-and-unit-labels. #16228 #16632 #16718 #16743
feature[FEATURE] PromQL: Add ts_of_(min|max|last)_over_time, behind feature flag experimental-promql-functions. #16722 #16733
feature[FEATURE] Scraping: Add global option always_scrape_classic_histograms to scrape a classic histogram even if it is also exposed as native. #16452
feature[FEATURE] OTLP: New config options promote_all_resource_attributes and ignore_resource_attributes. #16426
feature[FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
feature[ENHANCEMENT] Hetzner SD: Add label_selector to filter servers. #16512
other[ENHANCEMENT] PromQL: support non-constant parameter in aggregations like quantile and topk. #16404
other[ENHANCEMENT] UI: Better total target count display when using keep_dropped_targets option. #16604
feature[ENHANCEMENT] UI: Add simple filtering on the /rules page. #16605
other[ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
other[ENHANCEMENT] UI: Clear search field on /targets page. #16567
other[ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
other[ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
other[PERF] PromQL: Reuse histogramStatsIterator where possible. #16686
other[PERF] PromQL: Reuse storage for custom bucket values for native histograms. #16565
other[PERF] UI: Optimize memoization and search debouncing on /targets page. #16589
fix[PERF] UI: Fix full-page re-rendering when opening status nav menu. #16590
other[PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #16365
other[PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
other[PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #16158
other[PERF] Labels: Simpler/faster stringlabels encoding. #16069
other[PERF] Scraping: Reload scrape pools concurrently. #16595 #16783
fix[BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
fix[BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
fix[BUGFIX] PromQL: Fix native histogram last_over_time. #16744
fix[BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
fix[BUGFIX] PromQL: Don't emit a value from histogram_fraction or histogram_quantile if classic and native histograms are present at the same timestamp. #16552
fix[BUGFIX] PromQL: Incorrect rounding of [1001ms] to [1s] and similar. #16478
fix[BUGFIX] PromQL: Fix inconsistent / sometimes negative histogram_count and histogram_sum. #16682
fix[BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
fix[BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
fix[BUGFIX] PromQL: Improve consistency of avg aggregation and avg_over_time. #16569 #16773
fix[BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
fix[BUGFIX] API: Add HTTP Vary: Origin header to responses to avoid cache poisoning. #16008
fix[BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
fix[BUGFIX] Remote-write: For Azure AD auth, allow empty client_id to suppport system assigned managed identity. #16421
fix[BUGFIX] Scraping: Fix rare memory corruption bug. #16623
fix[BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
fix[BUGFIX] OTLP: Default config not respected when otlp: block is unset. #16693
fix[BUGFIX] OTLP receiver: Fix default configuration not being respected if the otlp: block is unset in the config file. #16693
fix[BUGFIX] Parser: Add reproducer for a dangling-reference issue in parsers. #16633
other[CHANGE] Config: Make setting out-of-order native histograms feature (--enable-feature=ooo-native-histograms) a no-op. Out-of-order native histograms are now always enabled when out_of_order_time_window is greater than zero and --enable-feature=native-histograms is set. #16207
feature[FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #15850
feature[FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #16441
feature[FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #16249
feature[FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #16360
other[ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #16095
feature[ENHANCEMENT] TSDB: add prometheus_tsdb_wal_replay_unknown_refs_total and prometheus_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during WAL/WBL replay. #16166
feature[ENHANCEMENT] Scraping: Add config option for escaping scheme request. #16066
feature[ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #16226
other[ENHANCEMENT] Alerting: make batch size configurable (--alertmanager.notification-batch-size). #16254
other[PERF] Kubernetes SD: make endpointSlice discovery more efficient. #16433
fix[BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #16340
fix[BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #16218
fix[BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #16231
fix[BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #16242
fix[BUGFIX] Promtool: Add feature flags for PromQL features. #16443
fix[BUGFIX] Rules: correct logging of alert name & template data. #15093
fix[BUGFIX] PromQL: Use arithmetic mean for histogram_stddev() and histogram_stdvar() . #16444
fix[BUGFIX] Azure SD: Fix panic on malformed log message. #16434 #16210
fix[BUGFIX] Config: Update GOGC before loading TSDB. #16491
feature[FEATURE] PromQL: Implement idelta() and irate() for native histograms. #15853
feature[ENHANCEMENT] Scaleway SD: Add meta_scaleway_instance_public_ipv4_addresses and meta_scaleway_instance_public_ipv6_addresses labels. #14228
other[ENHANCEMENT] TSDB: Reduce locking while reloading blocks. #12920
feature[ENHANCEMENT] PromQL: Allow UTF-8 labels in label_replace(). #15974
other[ENHANCEMENT] Promtool: tsdb create-blocks-from openmetrics can now read from a Pipe. #16011
feature[ENHANCEMENT] Rules: Add support for anchors and aliases in rule files. #14957
other[ENHANCEMENT] Dockerfile: Make /prometheus writable. #16073
other[ENHANCEMENT] API: Include scrape pool name for dropped targets in /api/v1/targets. #16085
other[ENHANCEMENT] UI: Improve time formatting and copying of selectors. #15999 #16165
other[ENHANCEMENT] UI: Bring back vertical grid lines and graph legend series toggling instructions. #16163 #16164
other[ENHANCEMENT] Mixin: The cluster label can be customized using clusterLabel. #15826
other[PERF] TSDB: Optimize some operations on head chunks by taking shortcuts. #12659
other[PERF] TSDB & Agent: Reduce memory footprint during WL replay. #15778
other[PERF] Remote-Write: Reduce memory footprint during WAL replay. #16197
other[PERF] API: Reduce memory footprint during header parsing. #16001
other[PERF] Rules: Improve dependency evaluation, enabling better concurrency. #16039
other[PERF] Scraping: Improve scraping performance for native histograms. #15731
other[PERF] Scraping: Improve parsing of created timestamps. #16072
fix[BUGFIX] Scraping: Bump cache iteration after error to avoid false duplicate detections. #16174
fix[BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #16218
fix[BUGFIX] PromQL: Fix counter reset detection for native histograms. #15902 #15987
fix[BUGFIX] PromQL: Fix inconsistent behavior with an empty range. #15970
fix[BUGFIX] PromQL: Fix inconsistent annotation in quantile_over_time(). #16018
fix[BUGFIX] PromQL: Prevent label_join() from producing duplicates. #15975
fix[BUGFIX] PromQL: Ignore native histograms in scalar(), sort() and sort_desc(). #15964
breaking[BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895
fix[BUGFIX] Alerting: Consider alert relabeling when deciding whether alerts are dropped. #15979
fix[BUGFIX] Config: Set GoGC to the default value in case of an empty configuration. #16052
breaking[BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060
fix[BUGFIX] Scaleway SD: The public IP will no longer be set to __meta_meta_scaleway_instance_public_ipv4 if it is an IPv6 address. #14228
fix[BUGFIX] UI: Display the correct value of Alerting rules' keep_firing_for. #16211
fix[BUGFIX] Don't send Accept header escape=allow-utf-8 when metric_name_validation_scheme: legacy is configured. #16061
other[CHANGE] relabel: Replace actions can now use UTF-8 characters in targetLabel field. Note that $ or ${} will be expanded. This also apply to replacement field for LabelMap action. #15851
other[CHANGE] rulefmt: Rule names can use UTF-8 characters, except { and } characters (due to common mistake checks). #15851
feature[FEATURE] remote/otlp: Add feature flag otlp-deltatocumulative to support conversion from delta to cumulative. #15165
other[ENHANCEMENT] openstack SD: Discover Octavia loadbalancers. #15539
feature[ENHANCEMENT] scrape: Add metadata for automatic metrics to WAL for metadata-wal-records feature. #15837
other[ENHANCEMENT] promtool: Support linting of scrape interval, through lint option too-long-scrape-interval. #15719
feature[ENHANCEMENT] promtool: Add --ignore-unknown-fields option. #15706
other[ENHANCEMENT] ui: Make "hide empty rules" and hide empty rules" persistent #15807
feature[ENHANCEMENT] web/api: Add a limit parameter to /query and /query_range. #15552
feature[ENHANCEMENT] api: Add fields Node and ServerTime to /status. #15784
other[PERF] Scraping: defer computing labels for dropped targets until they are needed by the UI. #15261
fix[BUGFIX] remotewrite2: Fix invalid metadata bug for metrics without metadata. #15829
fix[BUGFIX] remotewrite2: Fix the unit field propagation. #15825
fix[BUGFIX] scrape: Fix WAL metadata for histograms and summaries. #15832
fix[BUGFIX] ui: Merge duplicate "Alerts page settings" sections. #15810
fix[BUGFIX] PromQL: Fix functions with histograms. #15711
security[SECURITY] upgrade golang.org/x/crypto to address reported CVE-2024-45337. #15691
other[CHANGE] Notifier: Increment prometheus_notifications_errors_total by the number of affected alerts rather than per batch. #15428
other[CHANGE] API: list rules field "groupNextToken:omitempty" renamed to "groupNextToken". #15400
other[ENHANCEMENT] OTLP translate: keep identifying attributes in target_info. #15448
feature[ENHANCEMENT] Paginate rule groups, add infinite scroll to rules within groups. #15677
other[ENHANCEMENT] TSDB: Improve calculation of space used by labels. #13880
feature[ENHANCEMENT] Rules: new metric rule_group_last_rule_duration_sum_seconds. #15672
other[ENHANCEMENT] Observability: Export 'go_sync_mutex_wait_total_seconds_total' metric. #15339
other[ENHANCEMEN] Remote-Write: optionally use a DNS resolver that picks a random IP. #15329
other[PERF] Optimize l=~".+" matcher. #15474, #15684
other[PERF] TSDB: Cache all symbols for compaction . #15455
other[PERF] TSDB: MemPostings: keep a map of label values slices. #15426
breaking[PERF] Remote-Write: Remove interning hook. #15456
feature[PERF] Storage: Implement limit in mergeGenericQuerier. #14489
fix[BUGFIX] Main: Avoid possible segfault at exit. (#15724)
fix[BUGFIX] Rules: Do not run rules concurrently if uncertain about dependencies. #15560
fix[BUGFIX] PromQL: Adds test for absent, absent_over_time and deriv func with histograms. #15667
fix[BUGFIX] PromQL: Fix various bugs related to quoting UTF-8 characters. #15531
fix[BUGFIX] Scrape: fix nil panic after scrape loop reload. #15563
fix[BUGFIX] Remote-write: fix panic on repeated log message. #15562
fix[BUGFIX] Scrape: reload would ignore always_scrape_classic_histograms and convert_classic_histograms_to_nhcb configs. #15489
fix[BUGFIX] TSDB: fix data corruption in experimental native histograms. #15482
fix[BUGFIX] PromQL: Ignore histograms in all time related functions. #15479
fix[BUGFIX] OTLP receiver: Convert metric metadata. #15416
fix[BUGFIX] PromQL: Fix resets function for histograms. #15527
fix[BUGFIX] PromQL: Fix behaviour of changes() for mix of histograms and floats. #15469
fix[BUGFIX] PromQL: Fix behaviour of some aggregations with histograms. #15432
fix[BUGFIX] allow quoted exemplar keys in openmetrics text format. #15260
fix[BUGFIX] TSDB: fixes for rare conditions when loading write-behind-log (WBL). #15380
fix[BUGFIX] round() function did not remove name label. #15250
fix[BUGFIX] Promtool: analyze block shows metric name with 0 cardinality. #15438
fix[BUGFIX] PromQL: Fix count_values for histograms. #15422
fix[BUGFIX] PromQL: fix issues with comparison binary operations with bool modifier and native histograms. #15413
fix[BUGFIX] PromQL: fix incorrect "native histogram ignored in aggregation" annotations. #15414
fix[BUGFIX] PromQL: Corrects the behaviour of some operator and aggregators with Native Histograms. #15245
fix[BUGFIX] TSDB: Always return unknown hint for first sample in non-gauge histogram chunk. #15343
fix[BUGFIX] PromQL: Clamp functions: Ignore any points with native histograms. #15169
fix[BUGFIX] TSDB: Fix race on stale values in headAppender. #15322
fix[BUGFIX] UI: Fix selector / series formatting for empty metric names. #15340
fix[BUGFIX] OTLP receiver: Allow colons in non-standard units. #15710
+5 more items in the release notesfix[BUGFIX] Promql: Make subqueries left open. #15431
fix[BUGFIX] Fix memory leak when query log is enabled. #15434
fix[BUGFIX] Support utf8 names on /v1/label/:name/values endpoint. #15399
breaking[CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed. #15376
breaking[CHANGE] Set the GOMEMLIMIT variable automatically to match the Linux container memory limit. Use --no-auto-gomemlimit to disable it. The auto-gomemlimit feature flag was removed. #15373
breaking[CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a fallback_scrape_protocol in the scrape config. #15136
other[CHANGE] Remote-write: default enable_http2 to false. #15219
other[CHANGE] Scraping: normalize "le" and "quantile" label values upon ingestion. #15164
other[CHANGE] Scraping: config scrape_classic_histograms was renamed to always_scrape_classic_histograms. #15178
breaking[CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657
other[CHANGE] Disallow configuring AM with the v1 api. #13883
other[CHANGE] regexp . now matches all characters (performance improvement). #14505
other[CHANGE] holt_winters is now called double_exponential_smoothing and moves behind the experimental-promql-functions feature flag. #14930
other[CHANGE] API: The OTLP receiver endpoint can now be enabled using --web.enable-otlp-receiver instead of --enable-feature=otlp-write-receiver. #14894
feature[CHANGE] Prometheus will not add or remove port numbers from the target address. no-default-scrape-port feature flag removed. #14160
other[CHANGE] Logging: the format of log lines has changed a little, along with the adoption of Go's Structured Logging package. #14906
other[CHANGE] Don't create extra _created timeseries if feature-flag created-timestamp-zero-ingestion is enabled. #14738
other[CHANGE] Float literals and time durations being the same is now a stable fetaure. #15111
feature[CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch…
breaking[CHANGE] Kubernetes SD: Remove support for discovery.k8s.io/v1beta1 API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365
breaking[CHANGE] Kubernetes SD: Remove support for networking.k8s.io/v1beta1 API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365
breaking[CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed. #14705
breaking[CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807
breaking[CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed.…
breaking[CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead. #14747
deprecated[CHANGE] Remove deprecated remote-write-receiver,promql-at-modifier, and promql-negative-offset feature flags. #13456, #14526
deprecated[CHANGE] Remove deprecated storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention flags. #14640, #14643
feature[ENHANCEMENT] Tools: add debug printouts to promtool rules unit testing #15196
feature[ENHANCEMENT] PromQL: Introduce exponential interpolation for native histograms. #14677
feature[ENHANCEMENT] TSDB: Add support for ingestion of out-of-order native histogram samples. #14850, #14546
breaking[ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers. #13909
fix[BUGFIX] UI: fix selector / series formatting for empty metric names. #15341
fix[BUGFIX] PromQL: Fix stddev+stdvar aggregations to always ignore native histograms. #14941
fix[BUGFIX] PromQL: Fix stddev+stdvar aggregations to treat Infinity consistently. #14941
fix[BUGFIX] OTLP receiver: Preserve colons when generating metric names in suffix adding mode (this mode is always enabled, unless one uses Prometheus as a library). #15251
fix[BUGFIX] Scraping: Unit was missing when using protobuf format. #15095
fix[BUGFIX] PromQL: Only return "possible non-counter" annotation when rate returns points. #14910
fix[BUGFIX] TSDB: Chunks could have one unnecessary zero byte at the end. #14854
fix[BUGFIX] "superfluous response.WriteHeader call" messages in log. #14884
fix[BUGFIX] PromQL: Unary negation of native histograms. #14821
fix[BUGFIX] PromQL: Handle stale marker in native histogram series (e.g. if series goes away and comes back). #15025
fix[BUGFIX] Autoreload: Reload invalid yaml files. #14947
fix[BUGFIX] Scrape: Do not override target parameter labels with config params. #11029
+14 more items in the release notesfix[BUGFIX] round() function did not remove name label. #15250
feature[FEATURE] PromQL: Add experimental info function. #14495
other[FEATURE] Support UTF-8 characters in label names - feature flag utf8-names. #14482, #14880, #14736, #14727
feature[FEATURE] Scraping: Add the ability to set custom http_headers in config. #14817
other[FEATURE] Scraping: Support feature flag created-timestamp-zero-ingestion in OpenMetrics. #14356, #14815
other[FEATURE] Scraping: scrape_failure_log_file option to log failures to a file. #14734
other[FEATURE] OTLP receiver: Optional promotion of resource attributes to series labels. #14200
other[FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
feature[FEATURE] Promtool: tsdb create-blocks new option to add labels. #14403
feature[FEATURE] Promtool: promtool test adds --junit flag to format results. #14506
feature[FEATURE] TSDB: Add delayed-compaction feature flag, for people running many Prometheus to randomize timing. #12532
other[ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
other[ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
other[ENHANCEMENT] Remote Read client: Enable streaming remote read if the server supports it. #11379
other[ENHANCEMENT] Remote-Write: Don't reshard if we haven't successfully sent a sample since last update. #14450
other[ENHANCEMENT] PromQL: Delay deletion of name label to the end of the query evaluation. This is experimental and enabled under the feature-flag promql-delayed-name-removal. #14477
other[ENHANCEMENT] PromQL: Experimental sort_by_label and sort_by_label_desc sort by all labels when label is equal. #14655, #14985
other[ENHANCEMENT] PromQL: Clarify error message logged when Go runtime panic occurs during query evaluation. #14621
other[ENHANCEMENT] PromQL: Use Kahan summation for better accuracy in avg and avg_over_time. #14413
other[ENHANCEMENT] Tracing: Improve PromQL tracing, including showing the operation performed for aggregates, operators, and calls. #14816
other[ENHANCEMENT] API: Support multiple listening addresses. #14665
other[ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
other[PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948, #15120
other[PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
fix[BUGFIX] PromQL: make sort_by_label stable. #14985
fix[BUGFIX] SD: Fix dropping targets (with feature flag new-service-discovery-manager). #13147
fix[BUGFIX] SD: Stop storing stale targets (with feature flag new-service-discovery-manager). #13622
fix[BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
fix[BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
fix[BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
fix[BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14513, #14575, #14598, #14609, #14611, #14771, #14821
fix[BUGFIX] TSDB: Only count unknown record types in record_decode_failures_total metric. #14042
fix[BUGFIX] Scraping: allow multiple samples on same series, with explicit timestamps. #14685
fix[BUGFIX] Docker SD: fix crash in match_first_network mode when container is reconnected to a new network. #14654
fix[BUGFIX] PromQL: fix experimental native histogram counter reset detection on stale samples. #14514
fix[BUGFIX] PromQL: fix experimental native histograms getting corrupted due to vector selector bug in range queries. #14538
fix[BUGFIX] PromQL: fix experimental native histogram memory corruption when using histogram_count or histogram_sum. #14605
other[CHANGE] Remote-Write: highest_timestamp_in_seconds and queue_highest_sent_timestamp_seconds metrics now initialized to 0. #14437
other[CHANGE] API: Split warnings from info annotations in API response. #14327
other[FEATURE] Remote-Write: Version 2.0 experimental, plus metadata in WAL via feature flag metadata-wal-records (defaults on). #14395,#14427,#14444
feature[FEATURE] PromQL: add limitk() and limit_ratio() aggregation operators. #12503
other[ENHANCEMENT] PromQL: Accept underscores in literal numbers, e.g. 1_000_000 for 1 million. #12821
other[ENHANCEMENT] PromQL: float literal numbers and durations are now interchangeable (experimental). Example: time() - my_timestamp > 10m. #9138
other[ENHANCEMENT] PromQL: use Kahan summation for sum(). #14074,#14362
other[ENHANCEMENT] PromQL (experimental native histograms): Optimize histogram_count and histogram_sum functions. #14097
feature[ENHANCEMENT] TSDB: Better support for out-of-order experimental native histogram samples. #14438
other[ENHANCEMENT] TSDB: Optimise seek within index. #14393
other[ENHANCEMENT] TSDB: Optimise deletion of stale series. #14307
other[ENHANCEMENT] TSDB: Reduce locking to optimise adding and removing series. #13286,#14286
other[ENHANCEMENT] TSDB: Small optimisation: streamline special handling for out-of-order data. #14396,#14584
other[ENHANCEMENT] Regexps: Optimize patterns with multiple prefixes. #13843,#14368
other[ENHANCEMENT] Regexps: Optimize patterns containing multiple literal strings. #14173
other[ENHANCEMENT] AWS SD: expose Primary IPv6 addresses as __meta_ec2_primary_ipv6_addresses. #14156
feature[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks. #10490
other[ENHANCEMENT] OpenStack SD: Use flavor.original_name if available. #14312
other[ENHANCEMENT] UI (experimental native histograms): more accurate representation. #13680,#14430
other[ENHANCEMENT] Agent: out_of_order_time_window config option now applies to agent. #14094
other[ENHANCEMENT] Notifier: Send any outstanding Alertmanager notifications when shutting down. #14290
feature[ENHANCEMENT] Rules: Add label-matcher support to Rules API. #10194
feature[ENHANCEMENT] HTTP API: Add url to message logged on error while sending response. #14209
fix[BUGFIX] CLI: escape | characters when generating docs. #14420
fix[BUGFIX] PromQL (experimental native histograms): Fix some binary operators between native histogram values. #14454
fix[BUGFIX] TSDB: LabelNames API could fail during compaction. #14279
fix[BUGFIX] TSDB: Fix rare issue where pending OOO read can be left dangling if creating querier fails. #14341
fix[BUGFIX] TSDB: fix check for context cancellation in LabelNamesFor. #14302
fix[BUGFIX] Rules: Fix rare panic on reload. #14366
fix[BUGFIX] Config: In YAML marshalling, do not output a regexp field if it was never set. #14004
fix[BUGFIX] Remote-Write: reject samples with future timestamps. #14304
fix[BUGFIX] Remote-Write: Fix data corruption in remote write if max_sample_age is applied. #14078
fix[BUGFIX] Notifier: Fix Alertmanager discovery not updating under heavy load. #14174
fix[BUGFIX] Regexes: some Unicode characters were not matched by case-insensitive comparison. #14170,#14299
fix[BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515
feature[ENHANCEMENT] TSDB: Add backward compatibility with the upcoming TSDB block index v3 #16762
fix[BUGFIX] Top-level: Update GOGC before loading TSDB #16521
fix[BUGFIX] Runtime: fix GOGC is being set to 0 when installed with empty prometheus.yml file resulting high cpu usage. #16090
fix[BUGFIX] Scrape: fix dropping valid metrics after previous scrape failed. #16220
fix[BUGFIX] Scraping: allow multiple samples on same series, with explicit timestamps. #14685, #14740
fixFix a bug where Prometheus would crash with a segmentation fault if a remote-read request accessed a block on disk at about the same time as TSDB created a new block.
fix[BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515,#14523
fix[BUGFIX] Remote-write: stop dropping samples in catch-up #14446
other[CHANGE] Rules: Execute 1 query instead of N (where N is the number of alerts within alert rule) when restoring alerts. #13980 #14048
other[CHANGE] Runtime: Change GOGC threshold from 100 to 75 #14176 #14285
feature[FEATURE] Rules: Add new option query_offset for each rule group via rule group configuration file and rule_query_offset as part of the global configuration to have more resilience for remote write delays. #14061 #14216 #14273
feature[ENHANCEMENT] Rules: Add rule_group_last_restore_duration_seconds metric to measure the time it takes to restore a rule group. #13974
other[ENHANCEMENT] OTLP: Improve remote write format translation performance by using label set hashes for metric identifiers instead of string based ones. #14006 #13991
other[ENHANCEMENT] TSDB: Optimize querying with regexp matchers. #13620
fix[BUGFIX] OTLP: Don't generate target_info unless there are metrics and at least one identifying label is defined. #13991
fix[BUGFIX] Scrape: Do no try to ingest native histograms when the native histograms feature is turned off. This happened when protobuf scrape was enabled by for example the created time feature. #13987
fix[BUGFIX] Scaleway SD: Use the instance's public IP if no private IP is available as the address meta label. #13941
fix[BUGFIX] Query logger: Do not leak file descriptors on error. #13948
fix[BUGFIX] TSDB: Let queries with heavy regex matches be cancelled and not use up the CPU. #14096 #14103 #14118 #14199
fix[BUGFIX] API: Do not warn if result count is equal to the limit, only when exceeding the limit for the series, label-names and label-values APIs. #14116
fix[BUGFIX] TSDB: Fix head stats and hooks when replaying a corrupted snapshot. #14079
fix[CHANGE] TSDB: Fix the predicate checking for blocks which are beyond the retention period to include the ones right at the retention boundary. #9633
feature[FEATURE] Kubernetes SD: Add a new metric prometheus_sd_kubernetes_failures_total to track failed requests to Kubernetes API. #13554
feature[FEATURE] Kubernetes SD: Add node and zone metadata labels when using the endpointslice role. #13935
feature[FEATURE] Azure SD/Remote Write: Allow usage of Azure authorization SDK. #13099
other[FEATURE] Alerting: Support native histogram templating. #13731
other[FEATURE] Linode SD: Support IPv6 range discovery and region filtering. #13774
other[ENHANCEMENT] PromQL: Performance improvements for queries with regex matchers. #13461
other[ENHANCEMENT] PromQL: Performance improvements when using aggregation operators. #13744
other[ENHANCEMENT] PromQL: Validate label_join destination label. #13803
other[ENHANCEMENT] Scrape: Increment prometheus_target_scrapes_sample_duplicate_timestamp_total metric on duplicated series during one scrape. #12933
other[ENHANCEMENT] TSDB: Many improvements in performance. #13742 #13673 #13782
other[ENHANCEMENT] TSDB: Pause regular block compactions if the head needs to be compacted (prioritize head as it increases memory consumption). #13754
other[ENHANCEMENT] Observability: Improved logging during signal handling termination. #13772
other[ENHANCEMENT] Observability: All log lines for drop series use "num_dropped" key consistently. #13823
other[ENHANCEMENT] Observability: Log chunk snapshot and mmaped chunk replay duration during WAL replay. #13838
other[ENHANCEMENT] Observability: Log if the block is being created from WBL during compaction. #13846
fix[BUGFIX] PromQL: Fix inaccurate sample number statistic when querying histograms. #13667
fix[BUGFIX] PromQL: Fix histogram_stddev and histogram_stdvar for cases where the histogram has negative buckets. #13852
fix[BUGFIX] PromQL: Fix possible duplicated label name and values in a metric result for specific queries. #13845
fix[BUGFIX] Scrape: Fix setting native histogram schema factor during scrape. #13846
fix[BUGFIX] TSDB: Fix counting of histogram samples when creating WAL checkpoint stats. #13776
fix[BUGFIX] TSDB: Fix cases of compacting empty heads. #13755
fix[BUGFIX] TSDB: Count float histograms in WAL checkpoint. #13844
fix[BUGFIX] Remote Read: Fix memory leak due to broken requests. #13777
fix[BUGFIX] API: Stop building response for /api/v1/series/ when the API request was cancelled. #13766
fix[BUGFIX] promtool: Fix panic on promtool tsdb analyze --extended when no native histograms are present. #13976
fixBugfix release.
fix[BUGFIX] Notifier: could hang when using relabeling on alerts #13861
fix[BUGFIX] PromQL: Re-instate validation of label_join destination label #13803
fix[BUGFIX] Scraping (experimental native histograms): Fix handling of the min bucket factor on sync of targets #13846
fix[BUGFIX] PromQL: Some queries could return the same series twice (library use only) #13845
other[CHANGE] Scraping: Do experimental timestamp alignment even if tolerance is bigger than 1% of scrape interval #13624, #13737
feature[FEATURE] Alerting: Relabel rules for AlertManagerConfig; allows routing alerts to different alertmanagers #12551, #13735
feature[FEATURE] API: add limit param to series, label-names and label-values APIs #13396
feature[FEATURE] UI (experimental native histograms): Add native histogram chart to Table view #13658
feature[FEATURE] Promtool: Add a "tsdb dump-openmetrics" to dump in OpenMetrics format. #13194
feature[FEATURE] PromQL (experimental native histograms): Add histogram_avg function #13467
other[ENHANCEMENT] Rules: Evaluate independent rules concurrently #12946, #13527
other[ENHANCEMENT] Scraping (experimental native histograms): Support exemplars #13488
other[ENHANCEMENT] Remote Write: Disable resharding during active retry backoffs #13562
feature[ENHANCEMENT] Observability: Add native histograms to latency/duration metrics #13681
feature[ENHANCEMENT] Observability: Add 'type' label to prometheus_tsdb_head_out_of_order_samples_appended_total #13607
other[ENHANCEMENT] API: Faster generation of targets into JSON #13469, #13484
other[ENHANCEMENT] Scraping, API: Use faster compression library #10782
other[ENHANCEMENT] OpenTelemetry: Performance improvements in OTLP parsing #13627
other[ENHANCEMENT] PromQL: Optimisations to reduce CPU and memory #13448, #13536
fix[BUGFIX] PromQL: Constrain extrapolation in rate() to half of sample interval #13725
fix[BUGFIX] Remote Write: Stop slowing down when a new WAL segment is created #13583, #13628
fix[BUGFIX] PromQL: Fix wrongly scoped range vectors with @ modifier #13559
fix[BUGFIX] Kubernetes SD: Pod status changes were not discovered by Endpoints service discovery #13337
fix[BUGFIX] Azure SD: Fix 'error: parameter virtualMachineScaleSetName cannot be empty' (#13702)
fix[BUGFIX] Remote Write: Fix signing for AWS sigv4 transport #13497
fix[BUGFIX] Observability: Exemplars emitted by Prometheus use "trace_id" not "traceID" #13589
fix[BUGFIX] API: Fix metadata API using wrong field names. #13633
other[CHANGE] Remote Write: Error storage.ErrTooOldSample is now generating HTTP error 400 instead of HTTP error 500. #13335
feature[FEATURE] Experimental: Add support for ingesting zeros as created timestamps. (enabled under the feature-flag created-timestamp-zero-ingestion). #12733 #13279
feature[FEATURE] Promtool: Add analyze histograms command. #12331
feature[FEATURE] TSDB/compaction: Add a way to enable overlapping compaction. #13282 #13393 #13398
feature[FEATURE] Add automatic memory limit handling. Activated using the feature flag. auto-gomemlimit #13395
feature[ENHANCEMENT] Promtool: allow specifying multiple matchers in promtool tsdb dump. #13296
feature[ENHANCEMENT] Kuma SD: Extend configuration to allow users to specify client ID. #13278
fix[BUGFIX] Scraping: Fix quality value in HTTP Accept header. #13313
fix[BUGFIX] UI: Fix usage of the function time() that was crashing. #13371
fix[BUGFIX] Azure SD: Fix SD crashing when it finds a VM scale set. #13578
breakingtsdb: remove unused option
fixchunks.Reader: Fix typo in ChunkOrIterable doc string
featureAllow non-default registry to be used for metrics of SD components
featurekuma_sd: Extend Kuma SD configuration to allow users to specify ClientId
featurescripts: Add minimal permissions to script golangci-lint.yml
fixbug: Fix quality value in accept header
fixFix reusing float histograms
fixui: fix handling of scalar and string in isHeatmapData
featuretsdb/{index,compact}: allow using custom postings encoding format
featureprotobuf: add unit parser
featureAdd analyze histograms command to promtool
featuretsdb: add enable overlapping compaction
fixtsdb/compact: fix passing merge func
featureallow setting multiple matchers to "promtool tsdb dump"
fixfix slice copy in 1.20
featureExamples: link to rate for new users
featureadd cluster variable to Overview dashboard
fixFix regressions introduced by #13242
featureAdd warnings for histogramRate applied with isCounter not matching counter/gauge histogram
breakingremove obsolete build tag
featureTestLabelValuesWithMatchers: Add test case
fixfix(discovery tests): allow requireUpdate util to timeout in discovery/file…
breakingRemove csmarchbanks from remote write owners
featureAdd more context cancellation check at evaluation time
featureAdd automatic memory limit handling
featureTestHeadLabelValuesWithMatchers: Add test case
featureLabels: Add DropMetricName function, used in PromQL
fixui: fix for deep linking in stacked graphs
fixotlptranslator/update-copy.sh: Fix sed command lines
fixFix last_over_time for native histograms
+79 more items in the release notesfix[BUGFIX] TSDB: Fixed a wrong q= value in scrape accept header #13313
feature[FEATURE] Promtool: Add --run flag promtool test rules command. #12206
feature[FEATURE] SD: Add support for NS records to DNS SD. #13219
feature[FEATURE] UI: Add heatmap visualization setting in the Graph tab, useful histograms. #13096 #13371
feature[FEATURE] Scraping: Add scrape_config.enable_compression (default true) to disable gzip compression when scraping the target. #13166
feature[FEATURE] PromQL: Add a promql-experimental-functions feature flag containing some new experimental PromQL functions. #13103 NOTE: More experimental functions might be added behind the same feature flag in the future. Added functions:
feature[FEATURE] SD: Add __meta_linode_gpus label to Linode SD. #13097
feature[FEATURE] API: Add exclude_alerts query parameter to /api/v1/rules to only return recording rules. #12999
feature[FEATURE] Scraping: Add ability to specify priority of scrape protocols to accept during scrape (e.g. to scrape Prometheus proto format for certain jobs). This can be changed by setting global.scrape_protocols and scrape_config.scrape_protocols. #12738
feature[ENHANCEMENT] SD: Added cache for Azure SD to avoid rate-limits. #12622
fix[BUGFIX] SD: Fixed Azure SD public IP reporting #13241
fix[BUGFIX] API: Fix inaccuracies in posting cardinality statistics. #12653
fix[BUGFIX] PromQL: Fix inaccuracies of histogram_quantile with classic histograms. #13153
fix[BUGFIX] TSDB: Fix rare fails or inaccurate queries with OOO samples. #13115
fix[BUGFIX] TSDB: Fix rare panics on append commit when exemplars are used. #13092
fix[BUGFIX] TSDB: Fix exemplar WAL storage, so remote write can send/receive samples before exemplars. #13113
fix[BUGFIX] Mixins: Fix url filter on remote write dashboards. #10721
fix[BUGFIX] PromQL/TSDB: Various fixes to float histogram operations. #12891 #12977 #12609 #13190 #13189 #13191 #13201 #13212 #13208
fix[BUGFIX] Promtool: Fix int32 overflow issues for 32-bit architectures. #12978
featureAdd created timestamps to prompb
featureAdded ability to specify scrape protocols to accept during HTTP content type negotiation.
featureAdd the OpenSSF Scorecard GitHub Action
featureAssign new code owners for prometheus-mixin
featureadd generic shrink function
featureremote_write: add a unit test to make sure the write client sends the extra http headers as expected
breakinghead.go: Remove an unneeded snapshot trigger that was moved in https:…
featureFollow-up to #13060: Add test to ensure staleness tracking
featurecodemirror-promql: Add request header to client
featureLinode SD: Add __meta_linode_gpus label
featureAdd feature flag for PromQL experimental functions.
featureAdd enable_compression scrape config option
featureAdded Caching of network interface for Azure
featureAdd benchmark for native histograms
featurepromql: add sort_by_label and sort_by_label_desc functions
featureAdded Bartek as the next release shepherd.
featureDiscovery: Add support for NS records to DNS Discovery
featureAdd '--run' flag to promtool test rules
featureAdd test of Go module versions
featureprotobuf: add unit to metric family message
featureFloatHistogram.Add/Sub: handle any schema change
featureAdd mad_over_time function
+138 more items in the release notesfix[BUGFIX] TSDB: Make the wlog watcher read segments synchronously when not tailing. #13224
fix[BUGFIX] Agent: Participate in notify calls (fixes slow down in remote write handling introduced in 2.45). #13223
other[CHANGE] Remote-write: respect Retry-After header on 5xx errors. #12677
feature[FEATURE] Alerting: Add AWS SigV4 authentication support for Alertmanager endpoints. #12774
feature[FEATURE] Promtool: Add support for histograms in the TSDB dump command. #12775
feature[FEATURE] PromQL: Add warnings (and annotations) to PromQL query results. #12152 #12982 #12988 #13012
feature[FEATURE] Remote-write: Add Azure AD OAuth authentication support for remote write requests. #12572
feature[ENHANCEMENT] Remote-write: Add a header to count retried remote write requests. #12729
other[ENHANCEMENT] TSDB: Improve query performance by re-using iterator when moving between series. #12757
other[ENHANCEMENT] UI: Move /targets page discovered labels to expandable section #12824
other[ENHANCEMENT] TSDB: Optimize WBL loading by not sending empty buffers over channel. #12808
other[ENHANCEMENT] TSDB: Reply WBL mmap markers concurrently. #12801
feature[ENHANCEMENT] Promtool: Add support for specifying series matchers in the TSDB analyze command. #12842
other[ENHANCEMENT] PromQL: Prevent Prometheus from overallocating memory on subquery with large amount of steps. #12734
feature[ENHANCEMENT] PromQL: Add warning when monotonicity is forced in the input to histogram_quantile. #12931
other[ENHANCEMENT] Scraping: Optimize sample appending by reducing garbage. #12939
other[ENHANCEMENT] Storage: Reduce memory allocations in queries that merge series sets. #12938
other[ENHANCEMENT] UI: Show group interval in rules display. #12943
other[ENHANCEMENT] Scraping: Save memory when scraping by delaying creation of buffer. #12953
feature[ENHANCEMENT] Agent: Allow ingestion of out-of-order samples. #12897
feature[ENHANCEMENT] Promtool: Improve support for native histograms in TSDB analyze command. #12869
feature[ENHANCEMENT] Scraping: Add configuration option for tracking staleness of scraped timestamps. #13060
fix[BUGFIX] SD: Ensure that discovery managers are properly canceled. #10569
fix[BUGFIX] TSDB: Fix PostingsForMatchers race with creating new series. #12558
fix[BUGFIX] TSDB: Fix handling of explicit counter reset header in histograms. #12772
fix[BUGFIX] SD: Validate HTTP client configuration in HTTP, EC2, Azure, Uyuni, PuppetDB, and Lightsail SDs. #12762 #12811 #12812 #12815 #12814 #12816
fix[BUGFIX] TSDB: Fix counter reset edgecases causing native histogram panics. #12838
fix[BUGFIX] TSDB: Fix duplicate sample detection at chunk size limit. #12874
fix[BUGFIX] Promtool: Fix errors not being reported in check rules command. #12715
fix[BUGFIX] TSDB: Avoid panics reported in logs when head initialization takes a long time. #12876
fix[BUGFIX] TSDB: Ensure that WBL is repaired when possible. #12406
fix[BUGFIX] Storage: Fix crash caused by incorrect mixed samples handling. #13055
fix[BUGFIX] TSDB: Fix compactor failures by adding min time to histogram chunks. #13062
fix[BUGFIX] TSDB: Fix counter reset edgecases causing native histogram panics. #12838
fix[BUGFIX] Fix duplicate sample detection at chunk size limit #12874
feature[FEATURE] Web: Add OpenTelemetry (OTLP) Ingestion endpoint. #12571 #12643
other[FEATURE] Scraping: Optionally limit detail on dropped targets, to save memory. #12647
other[ENHANCEMENT] TSDB: Write head chunks to disk in the background to reduce blocking. #11818
other[ENHANCEMENT] PromQL: Speed up aggregate and function queries. #12682
other[ENHANCEMENT] PromQL: More efficient evaluation of query with timestamp(). #12579
other[ENHANCEMENT] API: Faster streaming of Labels to JSON. #12598
other[ENHANCEMENT] Agent: Memory pooling optimisation. #12651
other[ENHANCEMENT] TSDB: Prevent storage space leaks due to terminated snapshots on shutdown. #12664
other[ENHANCEMENT] Histograms: Refactoring and optimisations. #12352 #12584 #12596 #12711 #12054
feature[ENHANCEMENT] Histograms: Add histogram_stdvar and histogram_stddev functions. #12614
feature[ENHANCEMENT] Remote-write: add http.resend_count tracing attribute. #12676
other[ENHANCEMENT] TSDB: Support native histograms in snapshot on shutdown. #12722
fix[BUGFIX] TSDB/Agent: ensure that new series get written to WAL on rollback. #12592
fix[BUGFIX] Scraping: fix infinite loop on exemplar in protobuf format. #12737
feature[FEATURE] Promtool: Add PromQL format and label matcher set/delete commands to promtool. #11411
feature[FEATURE] Promtool: Add push metrics command. #12299
other[ENHANCEMENT] Promtool: Read from stdin if no filenames are provided in check rules. #12225
other[ENHANCEMENT] Hetzner SD: Support larger ID's that will be used by Hetzner in September. #12569
feature[ENHANCEMENT] Kubernetes SD: Add more labels for endpointslice and endpoints role. #10914
feature[ENHANCEMENT] Kubernetes SD: Do not add pods to target group if the PodIP status is not set. #11642
other[ENHANCEMENT] OpenStack SD: Include instance image ID in labels. #12502
other[ENHANCEMENT] Remote Write receiver: Validate the metric names and labels. #11688
other[ENHANCEMENT] Web: Initialize prometheus_http_requests_total metrics with code label set to 200. #12472
feature[ENHANCEMENT] TSDB: Add Zstandard compression option for wlog. #11666
other[ENHANCEMENT] TSDB: Support native histograms in snapshot on shutdown. #12258
other[ENHANCEMENT] Labels: Avoid compiling regexes that are literal. #12434
fix[BUGFIX] Histograms: Fix parsing of float histograms without zero bucket. #12577
fix[BUGFIX] Histograms: Fix scraping native and classic histograms missing some histograms. #12554
fix[BUGFIX] Histograms: Enable ingestion of multiple exemplars per sample. 12557
fix[BUGFIX] File SD: Fix path handling in File-SD watcher to allow directory monitoring on Windows. #12488
fix[BUGFIX] Linode SD: Cast InstanceSpec values to int64 to avoid overflows on 386 architecture. #12568
fix[BUGFIX] PromQL Engine: Include query parsing in active-query tracking. #12418
fix[BUGFIX] TSDB: Handle TOC parsing failures. #10623
securityThis release contains security fixes in dependencies and has been built with go1.21.11. #14309.
fix[BUGFIX] tsdb/agent: ensure that new series get written to WAL on rollback. #12592
fix[BUGFIX] Remote write: Avoid a race condition when applying configuration. #13135
fix[BUGFIX] Remote read: Release querier resources before encoding the results. #13777
breaking[BUGFIX] TSDB: Remove double memory snapshot on shutdown. #13110
fix[BUGFIX] TSDB: Fix PostingsForMatchers race with creating new series. #12558
other[ENHANCEMENT] Hetzner SD: Support larger ID's that will be used by Hetzner in September. #12619
fix[BUGFIX] Linode SD: Cast InstanceSpec values to int64 to avoid overflows on 386 architecture. #12568
fix[BUGFIX] TSDB: Handle TOC parsing failures. #10623
feature[FEATURE] API: New limit parameter to limit the number of items returned by /api/v1/status/tsdb endpoint. #12336
feature[FEATURE] Config: Add limits to global config. #12126
feature[FEATURE] Consul SD: Added support for path_prefix. #12372
feature[FEATURE] Native histograms: Add option to scrape both classic and native histograms. #12350
feature[FEATURE] Native histograms: Added support for two more arithmetic operators avg_over_time and sum_over_time. #12262
other[FEATURE] Promtool: When providing the block id, only one block will be loaded and analyzed. #12031
feature[FEATURE] Remote-write: New Azure ad configuration to support remote writing directly to Azure Monitor workspace. #11944
other[FEATURE] TSDB: Samples per chunk are now configurable with flag storage.tsdb.samples-per-chunk. By default set to its former value 120. #12055
other[ENHANCEMENT] Native histograms: bucket size can now be limited to avoid scrape fails. #12254
other[ENHANCEMENT] TSDB: Dropped series are now deleted from the WAL sooner. #12297
fix[BUGFIX] Native histograms: ChunkSeries iterator now checks if a new sample can be appended to the open chunk. #12185
fix[BUGFIX] Native histograms: Fix Histogram Appender Appendable() segfault. #12357
fix[BUGFIX] Native histograms: Fix setting reset header to gauge histograms in seriesToChunkEncoder. #12329
fix[BUGFIX] TSDB: Tombstone intervals are not modified after Get() call. #12245
fix[BUGFIX] TSDB: Use path/filepath to set the WAL directory. #12349
other[CHANGE] Remote-write: Raise default samples per send to 2,000. #12203
other[FEATURE] Remote-read: Handle native histograms. #12085, #12192
other[FEATURE] Promtool: Health and readiness check of prometheus server in CLI. #12096
feature[FEATURE] PromQL: Add query_samples_total metric, the total number of samples loaded by all queries. #12251
other[ENHANCEMENT] Storage: Optimise buffer used to iterate through samples. #12326
other[ENHANCEMENT] Scrape: Reduce memory allocations on target labels. #12084
other[ENHANCEMENT] PromQL: Use faster heap method for topk() / bottomk(). #12190
feature[ENHANCEMENT] Rules API: Allow filtering by rule name. #12270
other[ENHANCEMENT] Native Histograms: Various fixes and improvements. #11687, #12264, #12272
other[ENHANCEMENT] UI: Search of scraping pools is now case-insensitive. #12207
feature[ENHANCEMENT] TSDB: Add an affirmative log message for successful WAL repair. #12135
fix[BUGFIX] TSDB: Block compaction failed when shutting down. #12179
fix[BUGFIX] TSDB: Out-of-order chunks could be ignored if the write-behind log was deleted. #12127