Prometheus 3.13.0 → 3.16.0
9 releases · 4 breaking changes · 11 security fixes · 0 advisories fixed
11security4breaking5deprecated26feature87fix
Breaking changes
- breaking [ENHANCEMENT] promtool: Add --allow-incompatible-block-duration so users can specify any max-block-duration instead of being restricted to the default Prometheus block durations. #19363 in 3.16.0
- breaking [BUGFIX] Discovery: delete the stale prometheus_sd_last_update_timestamp_seconds series for a config that is removed on reload. #19131 in 3.15.0
- breaking [BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326 in 3.15.0
- breaking [BUGFIX] Rules: Clean up stale rule_group_last_rule_duration_sum_seconds and rule_group_last_restore_duration_seconds series when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing /metrics cardinality over time. #19107 in 3.14.0
Security changes
- security [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys. in 3.15.1
- security [SECURITY] Build with Go 1.27.3 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template. in 3.15.1
- security [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617). in 3.15.1
- security [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys. in 3.13.5
- security [SECURITY] Build with Go 1.26.10 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template. in 3.13.5
- security [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617). in 3.13.5
- security [SECURITY] Bump OpenTelemetry to v1.45.0 to fix exporter endpoint URLs leaking into info logs (GO-2026-6505). in 3.13.5
- security [SECURITY] Bump google.golang.org/grpc to v1.83.1 to fix HTTP/2 DATA frame fragmentation memory exhaustion (GO-2026-6348). #19834 in 3.13.4
- security [SECURITY] UI: Update vulnerable npm dependencies. #19834 in 3.13.4
- security [SECURITY] Bump github.com/klauspost/compress to v1.18.7 (GO-2026-5841) and golang.org/x/crypto to v0.55.0 (GO-2026-6303). in 3.13.3
- security [SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama in 3.13.2
Deprecations
- deprecated [CHANGE] Remote write: Stop exposing the deprecated prometheus_remote_storage_metadata_total, prometheus_remote_storage_metadata_failed_total, prometheus_remote_storage_metadata_retried_total, and prometheus_remote_storage_metadata_bytes_total metrics for Remote Write 2.0 queues. #20080 in 3.16.0
- deprecated [CHANGE] Logging: Deprecate --log.level; use runtime.log_level configuration to supply the default level. #19511 in 3.15.0
- deprecated [ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. --enable-feature=xor2-encoding is deprecated; use storage.tsdb.chunk_encoding.floats: xor2 instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461 in 3.15.0
- deprecated [BUGFIX] discovery/kubernetes: Populate __meta_kubernetes_service_loadbalancer_ip from status.loadBalancer.ingress, falling back to deprecated spec.loadBalancerIP. #19404 in 3.15.0
- deprecated [CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 in 3.14.0
Every release in between
deprecated[CHANGE] Remote write: Stop exposing the deprecated prometheus_remote_storage_metadata_total, prometheus_remote_storage_metadata_failed_total, prometheus_remote_storage_metadata_retried_total, and prometheus_remote_storage_metadata_bytes_total metrics for Remote Write 2.0 queues. #20080
feature[FEATURE] API: Allow overriding PromQL Start Timestamp processing on a per-query basis using the X-Prometheus-Use-Start-Timestamps HTTP header. #19066
feature[FEATURE] PromQL: Add experimental integral() function for computing integral-over-time of range vectors behind the --enable-feature=promql-experimental-functions flag. #14440
feature[FEATURE] PromQL: Add experimental query cost estimation, actual cost reporting, and configurable execution limits, enabled with --enable-feature=query-cost. #20028
feature[FEATURE] PromQL: Allow unquoted Unicode letters and dots behind a feature flag #20162
feature[ENHANCEMENT] Remote Read: Add native histogram prometheus_remote_read_handler_gate_wait_duration_seconds to measure concurrency-gate wait duration, including requests canceled while waiting. #18355
feature[ENHANCEMENT] Remote Write: Add experimental flag convert_nhcb_to_classic to convert NHCB to classic histograms for external systems that dont support NHCBs. #17205
breaking[ENHANCEMENT] promtool: Add --allow-incompatible-block-duration so users can specify any max-block-duration instead of being restricted to the default Prometheus block durations. #19363
fix[BUGFIX] Remote read: Abort failed streaming responses without writing duplicate headers or appending plain-text errors to the protobuf stream #19749
fix[BUGFIX] TSDB: Fix loss of out-of-order samples when a WAL checkpoint drops a series record that the out-of-order write-behind log still needs #19737
fix[BUGFIX] TSDB: Fix queries that seek within a head chunk returning samples committed after the query started #19854
fix[BUGFIX] TSDB: Preserve mapped chunks when WAL replay merges multiple references for the same series, preventing missing out-of-order samples and duplicate in-order chunks after restart #19805
fix[BUGFIX] Fix auto-reload logic to reload after reverting a failed configuration change #19838
fix[BUGFIX] API: Do not match metric family metadata for doubled _total and _info suffixes in experimental search API. #19910
fix[BUGFIX] API: Reject timeout and lookback_delta values that overflow int64 instead of turning them into a negative duration #19658
fix[BUGFIX] API: Resolve metric family metadata for suffixed metric names in search results, in experimental search API. #19824
fix[BUGFIX] AWS SD: Describe each ElastiCache cluster once instead of twice for clusters that are not replication group members. #19434
fix[BUGFIX] Build: Allow installing promtool with version-suffixed go install commands #19754
fix[BUGFIX] Discovery: Fix a panic in Azure service discovery when a VM tag has a null value. #19755
fix[BUGFIX] Federation: fix corruption of float native histograms. #19679
fix[BUGFIX] Fix ts_of_last_over_time for samples before the Unix epoch. #19828
fix[BUGFIX] Histograms: Fix ConvertNHCBToClassic inflating classic bucket counts for integer native histograms whose spans have gaps. #19659
fix[BUGFIX] Logging: Preserve rule and TSDB block values in JSON logs. #19572
fix[BUGFIX] OTLP receiver: Drop exemplars with timestamps more than 10 minutes in the future instead of storing them and blocking subsequent exemplars for that series. #19812
fix[BUGFIX] OTLP: Count only out-of-order exemplars in prometheus_api_otlp_out_of_order_exemplars_total, not every rejected exemplar. #20030
fix[BUGFIX] OTLP: Do not drop exemplars that have a trace_id or span_id filtered attribute in addition to a trace or span ID. The trace context now takes precedence over the colliding attribute, as required by the OpenTelemetry specification. #20004
fix[BUGFIX] OTLP: Do not emit le="+Inf" twice when a histogram's explicit bounds end at +Inf, and warn about the bound. #19425
fix[BUGFIX] OTLP: Stop dropping exemplars that arrive out of timestamp order within a data point. #20029
fix[BUGFIX] OpenMetrics 2.0: Attach histogram exemplars to matching classic _bucket series and preserve them when scraping both native and classic histograms. #20020
fix[BUGFIX] PromQL: Avoid intermediate overflow in rad() and deg() conversions. #19901
fix[BUGFIX] PromQL: Consistently ignore and drop type and unit labels in without aggregations, absent(), info(), and binary operator fill matching. #19986
fix[BUGFIX] PromQL: Do not panic when printing the partial AST returned alongside a parse error. #19797
fix[BUGFIX] PromQL: Drop type and unit in binary operations so they are transparent to users, similar to how name is handled. #18680
fix[BUGFIX] PromQL: Fix ==, !=, changes() and count_values on native histograms giving wrong results inside histogram_count(), histogram_sum() and histogram_avg(). #20024
fix[BUGFIX] PromQL: Fix histogram_fraction returning wrong results for native histograms with custom buckets when a bucket boundary is exactly 0. #19657
fix[BUGFIX] PromQL: Fix histogram_quantile and histogram_quantiles interpolating within the wrong bucket for native histograms whose sum is NaN. #19973
fix[BUGFIX] PromQL: Fix lexing of subqueries whose range is a duration keyword expression, e.g. foo[range():1s]. #19729
fix[BUGFIX] PromQL: Fix warning and error locations for duration expressions min_of and max_of. #19742
fix[BUGFIX] PromQL: Ignore NaN classic histogram bucket boundaries in histogram_quantile, histogram_quantiles and histogram_fraction and return correct value instead of NaN if there are normal buckets. #20057
fix[BUGFIX] PromQL: Preserve infinite values at exact quantile ranks. #19899
+45 more items in the release notessecurity[SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
security[SECURITY] Build with Go 1.27.3 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template.
security[SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
fix[CHANGE] PromQL: A range query whose end was not aligned to step caused subqueries inside it to evaluate past the parent's last actual step, inflating peakSamples in the query stats and against the query.max-samples limit, and wasting storage I/O reading samples that were never used in the result.…
deprecated[CHANGE] Logging: Deprecate --log.level; use runtime.log_level configuration to supply the default level. #19511
feature[FEATURE] Configuration: Allow changing the process log level through runtime.log_level on configuration reload. #19511
feature[FEATURE] Prometheus: Add --auto-gomemlimit.refresh-interval flag to periodically re-detect the container or system memory limit and update GOMEMLIMIT at runtime. #18843
feature[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. #12024. #18091
feature[FEATURE] scrape: Implement OM2.0 scrape format. #18606
feature[ENHANCEMENT] Docker SD: Add labels meta_docker_container_image and meta_docker_container_image_id. #19386
feature[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. #19500
deprecated[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. --enable-feature=xor2-encoding is deprecated; use storage.tsdb.chunk_encoding.floats: xor2 instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461
feature[ENHANCEMENT] TSDB: add prometheus_tsdb_head_appenders_created_total metric. #19411
feature[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. #19410
feature[ENHANCEMENT] storage/remote: Add undocumented failed_request_logging config field to debug log remote write V2 requests on send errors. #19249
feature[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. #18049
feature[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. #19332
feature[ENHANCEMENT] TSDB: Add prometheus_tsdb_head_series_pending_commit_underflow_total to report pending-sample reservation underflows. #19470
fix[BUGFIX] PromQL: Fix info() enrichment for composite expressions with mixed @/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. #19387
fix[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. #19557
fix[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. #19413
fix[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. #19272
fix[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. #19470
fix[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. #19470
fix[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. #19664
fix[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. #19460
fix[BUGFIX] TSDB: fix default block reload interval for custom options. #19368
fix[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. #19194
fix[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. #19435
fix[BUGFIX] AWS SD: Reject non-positive request_concurrency instead of hanging service discovery indefinitely. #19524
fix[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. #19302
fix[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. #19324
breaking[BUGFIX] Discovery: delete the stale prometheus_sd_last_update_timestamp_seconds series for a config that is removed on reload. #19131
fix[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. #19661
fix[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. #19438
fix[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. #19218
fix[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. #19444
fix[BUGFIX] Native histograms: DetectReset no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram rate()/increase() undercount. #19367
fix[BUGFIX] Never skip histogram buckets for histogram_stddev and histogram_stdvar functions. #19521
fix[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. #19343
fix[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. .\|(foo)\|.). #19516
breaking[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326
deprecated[BUGFIX] discovery/kubernetes: Populate __meta_kubernetes_service_loadbalancer_ip from status.loadBalancer.ingress, falling back to deprecated spec.loadBalancerIP. #19404
+40 more items in the release notesdeprecated[CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124
other[CHANGE] API: /api/v1/status/config now correctly shows separator: "" and replacement: "" in relabel configs when explicitly set to empty, instead of omitting them. #18653
other[CHANGE] Discovery/Hetzner: Drop the __meta_hetzner_datacenter label for hcloud targets, following its removal from the Hetzner Cloud API. #19269
other[CHANGE] PromQL: Enable duration expressions by default. The promql-duration-expr feature flag is now a no-op. #19033
other[CHANGE] PromQL: Promote first_over_time to stable. It no longer requires the promql-experimental-functions feature flag. #19093
feature[FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (oci_sd_configs). #18919
feature[FEATURE] PromQL: Add experimental start_timestamp(instant-vector) function returning the start timestamp of each sample in the given vector. Requires the use-start-timestamps feature flag. #19089
feature[FEATURE] PromQL: Allow rate() and increase() to use start timestamps as an alternative for rate extrapolation. Hidden behind the use-start-timestamps feature flag. #18619
feature[FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the histograms-st-encoding feature flag. #18609
other[ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g. k8s.pod.name and k8s_pod_name both become k8s_pod_name), and expose the prometheus_api_otlp_translation_warnings_total counter labelled by category to track such warnings.…
feature[ENHANCEMENT] Promtool: Add --remote-write.path flag to push metrics for backends that use a non-default remote-write endpoint. #19086
other[ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903
feature[ENHANCEMENT] TSDB: Add prometheus_tsdb_head_native_histogram_series and prometheus_tsdb_head_native_histogram_buckets gauges tracking the number of native histogram series and buckets in the head. #19170
feature[ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (step(), range(), min_of(), max_of()) in range selectors and subqueries. #18625
feature[ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706
other[ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181
other[PERF] Speed up regex label matchers matching a set of literal values (e.g. {job=~"foo|bar|baz"}). #18833
other[PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470
other[PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143
fix[BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
fix[BUGFIX] Discovery/AWS: Stop promtool check config from making AWS metadata service (IMDS) network calls when the region field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037
fix[BUGFIX] Discovery/Docker: Set a request timeout for docker_sd and dockerswarm_sd on unix, npipe, and tcp hosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237
fix[BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
fix[BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778
fix[BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
fix[BUGFIX] PromQL: Fix mad_over_time returning 0 instead of NaN when the range contains a NaN sample. #19040
fix[BUGFIX] Promtool: Accept --enable-feature=promql-binop-fill-modifiers in check rules, which previously rejected valid fill()/fill_left()/fill_right() expressions. #19153
fix[BUGFIX] Remote write: Respect the AZURE_FEDERATED_TOKEN_FILE environment variable for workload identity authentication instead of hardcoding the token file path. #18973
breaking[BUGFIX] Rules: Clean up stale rule_group_last_rule_duration_sum_seconds and rule_group_last_restore_duration_seconds series when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing /metrics cardinality over time. #19107
fix[BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
fix[BUGFIX] TSDB: Fix silent data loss and potential crash loop when stale_series_compaction_threshold is used in the config file. #19016
fix[BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted. #19016
fix[BUGFIX] TSDB: Fix prometheus_tsdb_head_stale_series over-counting and early eviction of series that change between float, integer histogram, and float histogram sample types. #19183
fix[BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
fix[BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
fix[BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
fix[BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart. #19202
fix[BUGFIX] TSDB: Surface query errors that were previously silently discarded. #19120
fix[BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with --enable-feature=xor2-encoding could silently revert to XOR after compaction. #19145
fix[BUGFIX] UI: Show the delete-series form on the TSDB Status page when --web.enable-admin-api is enabled. #19025
+2 more items in the release notessecurity[SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
security[SECURITY] Build with Go 1.26.10 to fix vulnerabilities in net/http, net/textproto, crypto/tls and html/template.
security[SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
security[SECURITY] Bump OpenTelemetry to v1.45.0 to fix exporter endpoint URLs leaking into info logs (GO-2026-6505).
other[CHANGE] Building Prometheus now requires Go 1.26 or later.
security[SECURITY] Bump google.golang.org/grpc to v1.83.1 to fix HTTP/2 DATA frame fragmentation memory exhaustion (GO-2026-6348). #19834
security[SECURITY] UI: Update vulnerable npm dependencies. #19834
fix[BUGFIX] Agent: Ignore unknown WAL record types to allow rolling back from newer versions. #19814
fix[BUGFIX] Federation: Fix corruption of float native histograms. #19679
fix[BUGFIX] Scrape: Fix failing scrapes of protobuf float histograms with zero sample, zero and bucket counts, such as the result of recording rate(x[1m]) over a constant histogram. #19682
security[SECURITY] Bump github.com/klauspost/compress to v1.18.7 (GO-2026-5841) and golang.org/x/crypto to v0.55.0 (GO-2026-6303).
fix[BUGFIX] Docker SD: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
fix[BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
fix[BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
fix[BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
fix[BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
fix[BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
fix[BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
security[SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama
fix[BUGFIX] PromQL: Preallocate the active query tracker file to avoid SIGBUS crashes when the data disk is full. #19289 by @akshajrawat
fix[BUGFIX] TSDB: Fix the head-chunk cache returning samples from the wrong chunk, or spurious not-found errors, to range queries after head-chunk truncation. #19134