Prometheus 2.51.1 → 3.11.1
40 releases · 19 breaking changes · 14 security fixes · 0 advisories fixed
14security19breaking6deprecated108feature240fix
Breaking changes
- breaking [ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. #17598 in 3.11.0
- breaking [BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. #18200 in 3.11.0
- breaking Remove the feature flag and make sure that everything still works as intended. in 3.8.0
- breaking [BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389 in 3.7.3
- breaking [BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344 in 3.7.1
- breaking [BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895 in 3.3.0
- breaking [BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060 in 3.3.0
- breaking [PERF] Remote-Write: Remove interning hook. #15456 in 3.1.0
- breaking [CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed. #15376 in 3.0.0
- breaking [CHANGE] Set the GOMEMLIMIT variable automatically to match the Linux container memory limit. Use --no-auto-gomemlimit to disable it. The auto-gomemlimit feature flag was removed. #15373 in 3.0.0
- breaking [CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a fallback_scrape_protocol in the scrape config. #15136 in 3.0.0
- breaking [CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657 in 3.0.0
- breaking [CHANGE] Kubernetes SD: Remove support for discovery.k8s.io/v1beta1 API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365 in 3.0.0
- breaking [CHANGE] Kubernetes SD: Remove support for networking.k8s.io/v1beta1 API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365 in 3.0.0
- breaking [CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed. #14705 in 3.0.0
- breaking [CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807 in 3.0.0
- breaking [CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed.… in 3.0.0
- breaking [CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead. #14747 in 3.0.0
- breaking [ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers. #13909 in 3.0.0
Security changes
- security [ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. #18347 in 3.11.0
- security [SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060 in 3.5.5
- security [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650 in 3.5.4
- security [SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934 in 3.5.4
- security [SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935 in 3.5.4
- security Shadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability. in 3.5.3
- security Brett Gervasoni for the AzureAD OAuth client_secret vulnerability. in 3.5.3
- security @iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability. in 3.5.3
- security [SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18587 in 3.5.3
- security [SECURITY] Remote-Write: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. #18591 in 3.5.3
- security [SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18585 in 3.5.3
- security [SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18589 in 3.5.3
- security [SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18507 in 3.5.2
- security [SECURITY] upgrade golang.org/x/crypto to address reported CVE-2024-45337. #15691 in 3.1.0
Deprecations
- deprecated [CHANGE] Hetzner SD: The meta_hetzner_datacenter label is deprecated for the role robot but kept for backward compatibility, use the meta_hetzner_robot_datacenter label instead. For the role hcloud, the label is deprecated and will stop working after the 1 July 2026. #17850 in 3.11.0
- deprecated [CHANGE] Hetzner SD: The meta_hetzner_hcloud_datacenter_location and meta_hetzner_hcloud_datacenter_location_network_zone labels are deprecated, use the meta_hetzner_hcloud_location and meta_hetzner_hcloud_location_network_zone labels instead. #17850 in 3.11.0
- deprecated [ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. #17829 in 3.11.0
- deprecated [CHANGE] Remote-write: the following metrics are deprecated: in 3.7.0
- deprecated [CHANGE] Remove deprecated remote-write-receiver,promql-at-modifier, and promql-negative-offset feature flags. #13456, #14526 in 3.0.0
- deprecated [CHANGE] Remove deprecated storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention flags. #14640, #14643 in 3.0.0
Every release in between
fix[BUGFIX] Tracing: Fix startup failure for OTLP HTTP tracing with insecure: true. #18469
deprecated[CHANGE] Hetzner SD: The meta_hetzner_datacenter label is deprecated for the role robot but kept for backward compatibility, use the meta_hetzner_robot_datacenter label instead. For the role hcloud, the label is deprecated and will stop working after the 1 July 2026. #17850
deprecated[CHANGE] Hetzner SD: The meta_hetzner_hcloud_datacenter_location and meta_hetzner_hcloud_datacenter_location_network_zone labels are deprecated, use the meta_hetzner_hcloud_location and meta_hetzner_hcloud_location_network_zone labels instead. #17850
feature[FEATURE] AWS SD: Add Elasticache Role. #18099
feature[FEATURE] AWS SD: Add RDS Role. #18206
feature[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. #17207
feature[FEATURE] Discovery: Introduce prometheus_sd_last_update_timestamp_seconds metric to track the last time a service discovery update was sent to consumers. #18194
feature[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. #18006
feature[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: meta_kubernetes_pod_deployment_name, meta_kubernetes_pod_cronjob_name and __meta_kubernetes_pod_job_name, respectively. #17774
feature[FEATURE] PromQL: Add / operators for trimming observations from native histograms. #17904
feature[FEATURE] PromQL: Add experimental histogram_quantiles variadic function for computing multiple quantiles at once. #17285
feature[FEATURE] TSDB: Add storage.tsdb.retention.percentage configuration to configure the maximum percent of disk usable for TSDB storage. #18080
feature[FEATURE] TSDB: Add an experimental st-storage feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. #18062
feature[FEATURE] TSDB: Add an experimental xor2-encoding feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. #18062
feature[ENHANCEMENT] HTTP client: Add AWS external_id support for sigv4. #17916
deprecated[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. #17829
breaking[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. #17598
feature[ENHANCEMENT] UI: Add autocomplete support for experimental first_over_time and ts_of_first_over_time PromQL functions. #18318
security[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. #18347
fix[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using KahanAdd. #18252
fix[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured endpoint option, a regression from the AWS SDK v2 migration. #18133
fix[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. #18133
fix[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. #17538
fix[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. #18244
fix[BUGFIX] Azure SD: Fix system-assigned managed identity not working when client_id is empty. #18323
fix[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. #17349
fix[BUGFIX] Kubernetes SD: Fix duplicate targets generated by *DualStack EndpointSlices policies. #18192
fix[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. #18084
fix[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. #18056
fix[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. #18263
fix[BUGFIX] Remote write: Fix prometheus_remote_storage_sent_batch_duration_seconds measuring before the request was sent. #18214
fix[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. #18375
fix[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. #18229
fix[BUGFIX] TSDB: Fail early when use-uncached-io feature flag is set on unsupported environments. #18219
breaking[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. #18200
fix[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. #17895
fix[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. #18306
fix[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. #18200
fix[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. #18189
fix[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. #18228
fix[BUGFIX] UI: Update retention display in runtime info when config is reloaded. #18200
+5 more items in the release notesfeature[CHANGE] Alerting: Add alertmanager dimension to following metrics: prometheus_notifications_dropped_total, prometheus_notifications_queue_capacity, prometheus_notifications_queue_length. #16355
other[CHANGE] UI: Hide expanded alert annotations by default, enabling more information density on the /alerts page. #17611
feature[FEATURE] AWS SD: Add MSK Role. #17600
feature[FEATURE] PromQL: Add fill() / fill_left() / fill_right() binop modifiers for specifying default values for missing series. #17644
feature[FEATURE] Web: Add OpenAPI 3.2 specification for the HTTP API at /api/v1/openapi.yaml. #17825
feature[FEATURE] Dockerfile: Add distroless image variant using UID/GID 65532 and no VOLUME declaration. Busybox image remains default. #17876
feature[FEATURE] Web: Add on-demand wall time profiling under /debug/pprof/fgprof. #18027
feature[ENHANCEMENT] PromQL: Add more detail to histogram quantile monotonicity info annotations. #15578
other[ENHANCEMENT] Alerting: Independent alertmanager sendloops. #16355
feature[ENHANCEMENT] TSDB: Experimental support for early compaction of stale series in the memory with configurable threshold stale_series_compaction_threshold in the config file. #16929
feature[ENHANCEMENT] Service Discovery: Service discoveries are now removable from the Prometheus binary through the Go build tag remove_all_sd and individual service discoveries can be re-added with the build tags enable__sd. Users can build a custom Prometheus with only the necessary SDs for a smaller…
other[ENHANCEMENT] Promtool: Support promql syntax features promql-duration-expr and promql-extended-range-selectors. #17926
other[PERF] PromQL: Avoid unnecessary label extraction in PromQL functions. #17676
other[PERF] PromQL: Improve performance of regex matchers like .-.-.*. #17707
feature[PERF] OTLP: Add label caching for OTLP-to-Prometheus conversion to reduce allocations and improve latency. #17860
other[PERF] API: Compute /api/v1/targets/relabel_steps in a single pass instead of re-running relabeling for each prefix. #17969
other[PERF] tsdb: Optimize LabelValues intersection performance for matchers. #18069
fix[BUGFIX] PromQL: Prevent query strings containing only UTF-8 continuation bytes from crashing Prometheus. #17735
fix[BUGFIX] Web: Fix missing X-Prometheus-Stopping header for /-/ready endpoint in NotReady state. #17795
fix[BUGFIX] PromQL: Fix PromQL info() function returning empty results when filtering by a label that exists on both the input metric and target_info. #17817
fix[BUGFIX] TSDB: Fix a bug during exemplar buffer grow/shrink that could cause exemplars to be incorrectly discarded. #17863
fix[BUGFIX] UI: Fix broken graph display after page reload, due to broken Y axis min encoding/decoding. #17869
fix[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields (Labels, Histogram pointers, metadata strings) before returning buffers to pools. #17879
fix[BUGFIX] PromQL: info function: fix series without identifying labels not being returned. #17898
fix[BUGFIX] OTLP: Filter name from OTLP attributes to prevent duplicate labels. #17917
fix[BUGFIX] TSDB: Fix division by zero when computing stale series ratio with empty head. #17952
fix[BUGFIX] OTLP: Fix potential silent data loss for sum metrics. #17954
fix[BUGFIX] PromQL: Fix smoothed interpolation across counter resets. #17988
fix[BUGFIX] PromQL: Fix panic with @ modifier on empty ranges. #18020
fix[BUGFIX] PromQL: Fix avg_over_time for a single native histogram. #18058
fix[BUGFIX] Agent: fix crash shortly after startup from invalid type of object. #17802
fix[BUGFIX] Scraping: fix relabel keep/drop not working. #17807
other[CHANGE] Native Histograms are no longer experimental! Make the native-histogram feature flag a no-op. Use scrape_native_histograms config option instead. #17528
feature[CHANGE] API: Add maximum limit of 10,000 sets of statistics to TSDB status endpoint. #17647
feature[FEATURE] API: Add /api/v1/features for clients to understand which features are supported. #17427
feature[FEATURE] Promtool: Add start_timestamp field for unit tests. #17636
feature[FEATURE] Promtool: Add --format seriesjson option to tsdb dump to output just series labels in JSON format. #13409
feature[FEATURE] Add --storage.tsdb.delay-compact-file.path flag for better interoperability with Thanos. #17435
feature[FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. #17714
feature[ENHANCEMENT]: TSDB: add flag --storage.tsdb.block-reload-interval to configure TSDB Block Reload Interval. #16728
feature[ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. #17565
other[ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. #16834
feature[ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. #17374
feature[ENHANCEMENT] Notifications: Add a histogram prometheus_notifications_latency_histogram_seconds to complement the existing summary. #16637
feature[ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. #17483
feature[ENHANCEMENT] SD: add a config label with job name for most prometheus_sd_refresh metrics. #17138
feature[ENHANCEMENT] TSDB: New histogram prometheus_tsdb_sample_ooo_delta, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. #17477
other[ENHANCEMENT] Remote-read: Validate histograms received via remote-read. #17561
other[PERF] TSDB: Small optimizations to postings index. #17439
other[PERF] Scraping: Speed up relabelling of series. #17530
other[PERF] PromQL: Small optimisations in binary operators. #17524, #17519.
fix[BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in _total. #17682
fix[BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. #17602
fix[BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. #17605
fix[BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. #17678
fix[BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty ignoring() and non-empty grouping. #17643
fix[BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. #17479
fix[BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. #17668
fix[BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. #17708
fix[BUGFIX] PromQL: Fix collision in label_join and label_replace with non-overlapping series. #17703
fix[BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. #17161
fix[BUGFIX] PromQL: Ensure that rate/increase/delta of histograms results in a gauge histogram. #17608
fix[BUGFIX] PromQL: Do not panic while iterating over invalid histograms. #17559
fix[BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. #17533
fix[BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. #17561
fix[BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. #17546
fix[BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. #17472
fix[BUGFIX] remote: Fix Remote Write receiver, so it does not send wrong response headers for v1 flow and cause Prometheus senders to emit false partial error log and metrics. #17683
otherUpgrade to v3.8 and keep the feature flag. Everything should work as before.
otherAt your own pace, set scrape_native_histograms to true in all relevant scrape configs. (There is a global and a per-scrape-config version of scrape_native_histograms, allowing granular control if needed. It is a good idea to also set scrape_native_histograms explicitly to false where you do not…
breakingRemove the feature flag and make sure that everything still works as intended.
otherNow you are ready for an upgrade to the next release (v3.9).
other[CHANGE] Remote-write 2 (receiving): Update to 2.0-rc.4 spec. "created timestamp" (CT) is now called "start timestamp" (ST). #17411
other[CHANGE] TSDB: Native Histogram Custom Bounds with a NaN threshold are now rejected. #17287
other[FEATURE] OAuth2: support jwt-bearer grant-type (RFC7523 3.1). #17592
feature[FEATURE] Dockerfile: Add OpenContainers spec labels to Dockerfile. #16483
feature[FEATURE] SD: Add unified AWS service discovery for ec2, lightsail and ecs services. #17046
other[FEATURE] Native histograms are now a stable, but optional feature, use the scrape_native_histograms config setting. #17232 #17315
other[FEATURE] UI: Support anchored and smoothed keyword in promql editor. #17239
other[FEATURE] UI: Show detailed relabeling steps for each discovered target. #17337
feature[FEATURE] Alerting: Add urlQueryEscape to template functions. #17403
feature[FEATURE] Promtool: Add Remote-Write 2.0 support to promtool push metrics via the --protobuf_message flag. #17417
other[ENHANCEMENT] Clarify the docs about handling negative native histograms. #17249
feature[ENHANCEMENT] Mixin: Add static UID to the remote-write dashboard. #17256
feature[ENHANCEMENT] PromQL: Reconcile mismatched NHCB bounds in Add and Sub. #17278
feature[ENHANCEMENT] Alerting: Add "unknown" state for alerting rules that haven't been evaluated yet. #17282
feature[ENHANCEMENT] Scrape: Allow simultaneous use of classic histogram → NHCB conversion and zero-timestamp ingestion. #17305
feature[ENHANCEMENT] UI: Add smoothed/anchored in explain. #17334
other[ENHANCEMENT] OTLP: De-duplicate any target_info samples with the same timestamp for the same series. #17400
other[ENHANCEMENT] Document use_fips_sts_endpoint in sigv4 config sections. #17304
other[ENHANCEMENT] Document Prometheus Agent. #14519
other[PERF] PromQL: Speed up parsing of variadic functions. #17316
other[PERF] UI: Speed up alerts/rules/... pages by not rendering collapsed content. #17485
other[PERF] UI: Performance improvement when getting label name and values in promql editor. #17194
other[PERF] UI: Speed up /alerts for many firing alerts via virtual scrolling. #17254
fix[BUGFIX] PromQL: Fix slice indexing bug in info function on churning series. #17199
fix[BUGFIX] API: Reduce lock contention on /api/v1/targets. #17306
fix[BUGFIX] PromQL: Consistent handling of gauge vs. counter histograms in aggregations. #17312
fix[BUGFIX] TSDB: Allow NHCB with -Inf as the first custom value. #17320
fix[BUGFIX] UI: Fix duplicate loading of data from the API speed up rendering of some pages. #17357
fix[BUGFIX] Old UI: Fix createExpressionLink to correctly build /graph URLs so links from Alerts/Rules work again. #17365
fix[BUGFIX] PromQL: Avoid panic when parsing malformed info call. #17379
fix[BUGFIX] PromQL: Include histograms when enforcing sample_limit. #17390
fix[BUGFIX] Config: Fix panic if TLS CA file is absent. #17418
fix[BUGFIX] PromQL: Fix histogram_fraction for classic histograms and NHCB if lower bound is in the first bucket. #17424
breaking[BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389
fix[BUGFIX] Fix federation of some native histograms. #17299 #17409
fix[BUGFIX] promtool: check config would fail when --lint=none flag was set. #17399 #17414
fix[BUGFIX] Remote-write: fix a deadlock in the queue resharding logic that can lead to suboptimal queue behavior. #17412
fix[BUGFIX] AWS SD: Fix AWS SDK v2 credentials handling for EC2 and Lightsail discovery. #17355
fix[BUGFIX] AWS SD: Load AWS region from IMDS when not set. #17376
fix[BUGFIX] Relabeling: Fix labelmap action validation with the legacy metric name validation scheme. #17372
fix[BUGFIX] PromQL: Fix parsing failure when anchored and smoothed are used as metric names and label names. #17353
fix[BUGFIX] PromQL: Fix formatting of range vector selectors with smoothed/anchored modifier. #17354
breaking[BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344
deprecated[CHANGE] Remote-write: the following metrics are deprecated:
otherprometheus_remote_storage_samples_in_total, use prometheus_wal_watcher_records_read_total{type="samples"} and prometheus_remote_storage_samples_dropped_total instead
otherprometheus_remote_storage_histograms_in_total, use prometheus_wal_watcher_records_read_total{type=~".*histogram_samples"} and prometheus_remote_storage_histograms_dropped_total instead
otherprometheus_remote_storage_exemplars_in_total, use prometheus_wal_watcher_records_read_total{type="exemplars"} and prometheus_remote_storage_exemplars_dropped_total instead
otherprometheus_remote_storage_highest_timestamp_in_seconds, use the more accurate prometheus_remote_storage_queue_highest_timestamp_seconds instead in dashboards and alerts to properly account for relabeling and for more accuracy. #17065
feature[FEATURE] PromQL: Add support for experimental anchored and smoothed rate behind feature flag promql-extended-range-selectors. #16457
feature[FEATURE] Federation: Add support for native histograms with custom buckets (NHCB). #17215
feature[FEATURE] PromQL: Add first_over_time(...) and ts_of_first_over_time(...) behind feature flag experimental-promql-functions. #16963 #17021
feature[FEATURE] Remote-write: Add support for Azure Workload Identity as an authentication method for the receiver. #16788
feature[FEATURE] Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the type-and-unit-labels feature flag is enabled. #17033
other[FEATURE] OTLP: Write start time of metrics as created time zero samples into TSDB when created-timestamp-zero-ingestion feature flag is enabled. #16951
feature[ENHANCEMENT] PromQL: Add warn-level annotations for counter reset conflicts in certain histogram operations. #17051 #17094
feature[ENHANCEMENT] UI: Add scrape interval and scrape timeout to targets page. #17158
other[ENHANCEMENT] TSDB: Reduce the resolution of native histograms read from chunks or remote read if the schema is exponential. #17213
feature[ENHANCEMENT] Remote write: Add logging for unexpected metadata in sample batches, when metadata entries are found in samples-only batches. #17034 #17082
other[ENHANCEMENT] Rules: Support concurrent evaluation for rules querying ALERTS and ALERTS_FOR_STATE. #17064
feature[ENHANCEMENT] TSDB: Add logs to improve visibility into internal operations. #17074
other[PERF] OTLP: Write directly to TSDB instead of passing through a Remote-Write adapter when receiving OTLP metrics. #16951
other[PERF] OTLP: Reduce number of logs emitted from OTLP endpoint. No need to log duplicate sample errors. #17201
other[PERF] PromQL: Move more work to preprocessing step. #16896
other[PERF] PromQL: Reduce allocations when walking the syntax tree. #16593
other[PERF] TSDB: Optimize appender creation, slightly speeding up startup. #16922
other[PERF] TSDB: Improve speed of querying a series with multiple matchers. #13971
fix[BUGFIX] Alerting: Mutating alerts relabeling (using replace actions, etc.) within a alertmanager_config.alert_relabel_configs block is now scoped correctly and no longer yields altered alerts to subsequent blocks. #17063
fix[BUGFIX] Config: Infer valid escaping scheme when scrape config validation scheme is set. #16923
fix[BUGFIX] TSDB: Correctly handle appending mixed-typed samples to the same series. #17071 #17241 #17290 #17295 #17296
fix[BUGFIX] Remote-write: Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0, log warning. #17146
fix[BUGFIX] TSDB: Fix metadata entries handling on metadata-wal-records experimental feature for native histograms with custom buckets (NHCB) in protobuf scraping. #17156
fix[BUGFIX] TSDB: Ignore Native Histograms with invalid schemas during WAL/WBL replay. #17214
fix[BUGFIX] PromQL: Avoid empty metric names in annotations for histogram_quantile(). #16794
fix[BUGFIX] PromQL: Correct inaccurate character positions in errors for some aggregate expressions. #16996 #17031
fix[BUGFIX] PromQL: Fix info() function on churning series. #17135
fix[BUGFIX] PromQL: Set native histogram to gauge type when subtracting or multiplying/dividing with negative factors. #17004
fix[BUGFIX] TSDB: Reject unsupported native histogram schemas when attempting to append to TSDB. For scrape and remote-write implement reducing the resolution to fit the maximum if the schema is within the -9 to 52. #17189
fix[BUGFIX] Remote-write: Fix HTTP handler to return after writing error response for invalid compression. #17050
fix[BUGFIX] Remote-write: Return HTTP error 400 instead of 5xx for wrongly formatted Native Histograms. #17210
fix[BUGFIX] Scrape: Prevent staleness markers from generating unnecessary series. #16429
fix[BUGFIX] TSDB: Avoid misleading Failed to calculate size of \"wal\" dir error logs during WAL clean-up. #17006
fix[BUGFIX] TSDB: Prevent erroneously dropping series records during WAL checkpoints. #17029
fix[BUGFIX] UI: Fix redirect to path of -web.external-url if -web.route-prefix is configured. #17240
+1 more item in the release notesfeature[FEATURE] PromQL: Add step(), and min() and max() on durations, behind feature flag promql-duration-expr. #16777
feature[FEATURE] API: Add a /v1/status/tsdb/blocks endpoint exposing metadata about loaded blocks. #16695
feature[FEATURE] Templates: Add toDuration() and now() functions. #16619
feature[ENHANCEMENT] Discovery: Add support for attaching namespace metadata to targets. #16831
feature[ENHANCEMENT] OTLP: Support new UnderscoreEscapingWithoutSuffixes strategy via otlp.translation_strategy. #16849
other[ENHANCEMENT] OTLP: Support including scope metadata as metric labels via otlp.promote_scope_metadata. #16878
feature[ENHANCEMENT] OTLP: Add type and unit labels when feature flag type-and-unit-labels is enabled. #16630
other[ENHANCEMENT] Tracing: Send the traceparent HTTP header during scrapes. #16425
feature[ENHANCEMENT] UI: Add option to disable info and warning query messages under Query page settings. #16901
other[ENHANCEMENT] UI: Improve metadata handling for _count/_sum/_bucket suffixes. #16910
other[ENHANCEMENT] TSDB: Track stale series in the Head block via the prometheus_tsdb_head_stale_series metric. #16925
other[PERF] PromQL: Improve performance due to internal optimizations. #16797
fix[BUGFIX] Config: Fix "unknown global name escaping method" error messages produced during config validation. #16801
fix[BUGFIX] Discovery: Fix race condition during shutdown. #16820
fix[BUGFIX] OTLP: Generate target_info samples between the earliest and latest samples per resource. #16737
fix[BUGFIX] PromQL: Fail when NaN is passed as parameter to topk(), bottomk(), limitk() and limit_ratio(). #16725
fix[BUGFIX] PromQL: Fix extrapolation for native counter histograms. #16828
fix[BUGFIX] PromQL: Reduce numerical errors by disabling some optimizations. #16895
fix[BUGFIX] PromQL: Fix inconsistencies when using native histograms in subqueries. #16879
fix[BUGFIX] PromQL: Fix inconsistent annotations for rate() and increase() on histograms when feature flag type-and-unit-labels is enabled. #16915
fix[BUGFIX] Scraping: Fix memory corruption in slicelabels builds. #16946
fix[BUGFIX] TSDB: Fix panic on append when feature flag created-timestamp-zero-ingestion is enabled. #16332
fix[BUGFIX] TSDB: Fix panic on append for native histograms with empty buckets. #16893
security[SECURITY] UI: Bump sanitize-html to v2.17.5 to fix CVE-2026-53606. #19060
security[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via /-/config endpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650
security[SECURITY] Dependencies: Bump golang.org/x/net to v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934
security[SECURITY] UI: Bump mantine-ui dependencies (react-router-dom, vitest, vite, postcss) to their patched versions to resolve security advisories. #18935
other[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18792
securityShadowbyte (4c1dr3aper) - Charlie Lewis for the Remote-Read snappy decode vulnerability.
securityBrett Gervasoni for the AzureAD OAuth client_secret vulnerability.
security@iiihaiii and @Ngocnn97 for the Old UI XSS vulnerability.
security[SECURITY] AzureAD remote write: Fix OAuth client_secret being exposed in plaintext via /-/config endpoint. GHSA-wg65-39gg-5wfj / CVE-2026-42151 #18587
security[SECURITY] Remote-Write: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. #18591
security[SECURITY] Remote-read: Reject snappy-compressed requests whose declared decoded length exceeds the decode limit. GHSA-8rm2-7qqf-34qm / CVE-2026-42154 #18585
security[SECURITY] UI: Fix stored XSS via unescaped le label values in old UI heatmap chart tick labels. GHSA-fw8g-cg8f-9j28 #18589
security[SECURITY] UI: Fix stored XSS via unescaped metric names and labels. CVE-2026-40179. #18507
other[PERF] Regex: Stop calling Simplify. #17908
otherDocker library updated from 28.2.2 to 28.5.2. #17821
otherBuilt with Go 1.24.11.
feature[FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag type-and-unit-labels. #16228 #16632 #16718 #16743
feature[FEATURE] PromQL: Add ts_of_(min|max|last)_over_time, behind feature flag experimental-promql-functions. #16722 #16733
feature[FEATURE] Scraping: Add global option always_scrape_classic_histograms to scrape a classic histogram even if it is also exposed as native. #16452
feature[FEATURE] OTLP: New config options promote_all_resource_attributes and ignore_resource_attributes. #16426
feature[FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
feature[ENHANCEMENT] Hetzner SD: Add label_selector to filter servers. #16512
other[ENHANCEMENT] PromQL: support non-constant parameter in aggregations like quantile and topk. #16404
other[ENHANCEMENT] UI: Better total target count display when using keep_dropped_targets option. #16604
feature[ENHANCEMENT] UI: Add simple filtering on the /rules page. #16605
other[ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
other[ENHANCEMENT] UI: Clear search field on /targets page. #16567
other[ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
other[ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
other[PERF] PromQL: Reuse histogramStatsIterator where possible. #16686
other[PERF] PromQL: Reuse storage for custom bucket values for native histograms. #16565
other[PERF] UI: Optimize memoization and search debouncing on /targets page. #16589
fix[PERF] UI: Fix full-page re-rendering when opening status nav menu. #16590
other[PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #16365
other[PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
other[PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #16158
other[PERF] Labels: Simpler/faster stringlabels encoding. #16069
other[PERF] Scraping: Reload scrape pools concurrently. #16595 #16783
fix[BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
fix[BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
fix[BUGFIX] PromQL: Fix native histogram last_over_time. #16744
fix[BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
fix[BUGFIX] PromQL: Don't emit a value from histogram_fraction or histogram_quantile if classic and native histograms are present at the same timestamp. #16552
fix[BUGFIX] PromQL: Incorrect rounding of [1001ms] to [1s] and similar. #16478
fix[BUGFIX] PromQL: Fix inconsistent / sometimes negative histogram_count and histogram_sum. #16682
fix[BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
fix[BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
fix[BUGFIX] PromQL: Improve consistency of avg aggregation and avg_over_time. #16569 #16773
fix[BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
fix[BUGFIX] API: Add HTTP Vary: Origin header to responses to avoid cache poisoning. #16008
fix[BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
fix[BUGFIX] Remote-write: For Azure AD auth, allow empty client_id to suppport system assigned managed identity. #16421
fix[BUGFIX] Scraping: Fix rare memory corruption bug. #16623
fix[BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
fix[BUGFIX] OTLP: Default config not respected when otlp: block is unset. #16693
fix[BUGFIX] OTLP receiver: Fix default configuration not being respected if the otlp: block is unset in the config file. #16693
fix[BUGFIX] Parser: Add reproducer for a dangling-reference issue in parsers. #16633
other[CHANGE] Config: Make setting out-of-order native histograms feature (--enable-feature=ooo-native-histograms) a no-op. Out-of-order native histograms are now always enabled when out_of_order_time_window is greater than zero and --enable-feature=native-histograms is set. #16207
feature[FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #15850
feature[FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #16441
feature[FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #16249
feature[FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #16360
other[ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #16095
feature[ENHANCEMENT] TSDB: add prometheus_tsdb_wal_replay_unknown_refs_total and prometheus_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during WAL/WBL replay. #16166
feature[ENHANCEMENT] Scraping: Add config option for escaping scheme request. #16066
feature[ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #16226
other[ENHANCEMENT] Alerting: make batch size configurable (--alertmanager.notification-batch-size). #16254
other[PERF] Kubernetes SD: make endpointSlice discovery more efficient. #16433
fix[BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #16340
fix[BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #16218
fix[BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #16231
fix[BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #16242
fix[BUGFIX] Promtool: Add feature flags for PromQL features. #16443
fix[BUGFIX] Rules: correct logging of alert name & template data. #15093
fix[BUGFIX] PromQL: Use arithmetic mean for histogram_stddev() and histogram_stdvar() . #16444
fix[BUGFIX] Azure SD: Fix panic on malformed log message. #16434 #16210
fix[BUGFIX] Config: Update GOGC before loading TSDB. #16491
feature[FEATURE] PromQL: Implement idelta() and irate() for native histograms. #15853
feature[ENHANCEMENT] Scaleway SD: Add meta_scaleway_instance_public_ipv4_addresses and meta_scaleway_instance_public_ipv6_addresses labels. #14228
other[ENHANCEMENT] TSDB: Reduce locking while reloading blocks. #12920
feature[ENHANCEMENT] PromQL: Allow UTF-8 labels in label_replace(). #15974
other[ENHANCEMENT] Promtool: tsdb create-blocks-from openmetrics can now read from a Pipe. #16011
feature[ENHANCEMENT] Rules: Add support for anchors and aliases in rule files. #14957
other[ENHANCEMENT] Dockerfile: Make /prometheus writable. #16073
other[ENHANCEMENT] API: Include scrape pool name for dropped targets in /api/v1/targets. #16085
other[ENHANCEMENT] UI: Improve time formatting and copying of selectors. #15999 #16165
other[ENHANCEMENT] UI: Bring back vertical grid lines and graph legend series toggling instructions. #16163 #16164
other[ENHANCEMENT] Mixin: The cluster label can be customized using clusterLabel. #15826
other[PERF] TSDB: Optimize some operations on head chunks by taking shortcuts. #12659
other[PERF] TSDB & Agent: Reduce memory footprint during WL replay. #15778
other[PERF] Remote-Write: Reduce memory footprint during WAL replay. #16197
other[PERF] API: Reduce memory footprint during header parsing. #16001
other[PERF] Rules: Improve dependency evaluation, enabling better concurrency. #16039
other[PERF] Scraping: Improve scraping performance for native histograms. #15731
other[PERF] Scraping: Improve parsing of created timestamps. #16072
fix[BUGFIX] Scraping: Bump cache iteration after error to avoid false duplicate detections. #16174
fix[BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #16218
fix[BUGFIX] PromQL: Fix counter reset detection for native histograms. #15902 #15987
fix[BUGFIX] PromQL: Fix inconsistent behavior with an empty range. #15970
fix[BUGFIX] PromQL: Fix inconsistent annotation in quantile_over_time(). #16018
fix[BUGFIX] PromQL: Prevent label_join() from producing duplicates. #15975
fix[BUGFIX] PromQL: Ignore native histograms in scalar(), sort() and sort_desc(). #15964
breaking[BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895
fix[BUGFIX] Alerting: Consider alert relabeling when deciding whether alerts are dropped. #15979
fix[BUGFIX] Config: Set GoGC to the default value in case of an empty configuration. #16052
breaking[BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060
fix[BUGFIX] Scaleway SD: The public IP will no longer be set to __meta_meta_scaleway_instance_public_ipv4 if it is an IPv6 address. #14228
fix[BUGFIX] UI: Display the correct value of Alerting rules' keep_firing_for. #16211
fix[BUGFIX] Don't send Accept header escape=allow-utf-8 when metric_name_validation_scheme: legacy is configured. #16061
other[CHANGE] relabel: Replace actions can now use UTF-8 characters in targetLabel field. Note that $ or ${} will be expanded. This also apply to replacement field for LabelMap action. #15851
other[CHANGE] rulefmt: Rule names can use UTF-8 characters, except { and } characters (due to common mistake checks). #15851
feature[FEATURE] remote/otlp: Add feature flag otlp-deltatocumulative to support conversion from delta to cumulative. #15165
other[ENHANCEMENT] openstack SD: Discover Octavia loadbalancers. #15539
feature[ENHANCEMENT] scrape: Add metadata for automatic metrics to WAL for metadata-wal-records feature. #15837
other[ENHANCEMENT] promtool: Support linting of scrape interval, through lint option too-long-scrape-interval. #15719
feature[ENHANCEMENT] promtool: Add --ignore-unknown-fields option. #15706
other[ENHANCEMENT] ui: Make "hide empty rules" and hide empty rules" persistent #15807
feature[ENHANCEMENT] web/api: Add a limit parameter to /query and /query_range. #15552
feature[ENHANCEMENT] api: Add fields Node and ServerTime to /status. #15784
other[PERF] Scraping: defer computing labels for dropped targets until they are needed by the UI. #15261
fix[BUGFIX] remotewrite2: Fix invalid metadata bug for metrics without metadata. #15829
fix[BUGFIX] remotewrite2: Fix the unit field propagation. #15825
fix[BUGFIX] scrape: Fix WAL metadata for histograms and summaries. #15832
fix[BUGFIX] ui: Merge duplicate "Alerts page settings" sections. #15810
fix[BUGFIX] PromQL: Fix functions with histograms. #15711
security[SECURITY] upgrade golang.org/x/crypto to address reported CVE-2024-45337. #15691
other[CHANGE] Notifier: Increment prometheus_notifications_errors_total by the number of affected alerts rather than per batch. #15428
other[CHANGE] API: list rules field "groupNextToken:omitempty" renamed to "groupNextToken". #15400
other[ENHANCEMENT] OTLP translate: keep identifying attributes in target_info. #15448
feature[ENHANCEMENT] Paginate rule groups, add infinite scroll to rules within groups. #15677
other[ENHANCEMENT] TSDB: Improve calculation of space used by labels. #13880
feature[ENHANCEMENT] Rules: new metric rule_group_last_rule_duration_sum_seconds. #15672
other[ENHANCEMENT] Observability: Export 'go_sync_mutex_wait_total_seconds_total' metric. #15339
other[ENHANCEMEN] Remote-Write: optionally use a DNS resolver that picks a random IP. #15329
other[PERF] Optimize l=~".+" matcher. #15474, #15684
other[PERF] TSDB: Cache all symbols for compaction . #15455
other[PERF] TSDB: MemPostings: keep a map of label values slices. #15426
breaking[PERF] Remote-Write: Remove interning hook. #15456
feature[PERF] Storage: Implement limit in mergeGenericQuerier. #14489
fix[BUGFIX] Main: Avoid possible segfault at exit. (#15724)
fix[BUGFIX] Rules: Do not run rules concurrently if uncertain about dependencies. #15560
fix[BUGFIX] PromQL: Adds test for absent, absent_over_time and deriv func with histograms. #15667
fix[BUGFIX] PromQL: Fix various bugs related to quoting UTF-8 characters. #15531
fix[BUGFIX] Scrape: fix nil panic after scrape loop reload. #15563
fix[BUGFIX] Remote-write: fix panic on repeated log message. #15562
fix[BUGFIX] Scrape: reload would ignore always_scrape_classic_histograms and convert_classic_histograms_to_nhcb configs. #15489
fix[BUGFIX] TSDB: fix data corruption in experimental native histograms. #15482
fix[BUGFIX] PromQL: Ignore histograms in all time related functions. #15479
fix[BUGFIX] OTLP receiver: Convert metric metadata. #15416
fix[BUGFIX] PromQL: Fix resets function for histograms. #15527
fix[BUGFIX] PromQL: Fix behaviour of changes() for mix of histograms and floats. #15469
fix[BUGFIX] PromQL: Fix behaviour of some aggregations with histograms. #15432
fix[BUGFIX] allow quoted exemplar keys in openmetrics text format. #15260
fix[BUGFIX] TSDB: fixes for rare conditions when loading write-behind-log (WBL). #15380
fix[BUGFIX] round() function did not remove name label. #15250
fix[BUGFIX] Promtool: analyze block shows metric name with 0 cardinality. #15438
fix[BUGFIX] PromQL: Fix count_values for histograms. #15422
fix[BUGFIX] PromQL: fix issues with comparison binary operations with bool modifier and native histograms. #15413
fix[BUGFIX] PromQL: fix incorrect "native histogram ignored in aggregation" annotations. #15414
fix[BUGFIX] PromQL: Corrects the behaviour of some operator and aggregators with Native Histograms. #15245
fix[BUGFIX] TSDB: Always return unknown hint for first sample in non-gauge histogram chunk. #15343
fix[BUGFIX] PromQL: Clamp functions: Ignore any points with native histograms. #15169
fix[BUGFIX] TSDB: Fix race on stale values in headAppender. #15322
fix[BUGFIX] UI: Fix selector / series formatting for empty metric names. #15340
fix[BUGFIX] OTLP receiver: Allow colons in non-standard units. #15710
+5 more items in the release notesfix[BUGFIX] Promql: Make subqueries left open. #15431
fix[BUGFIX] Fix memory leak when query log is enabled. #15434
fix[BUGFIX] Support utf8 names on /v1/label/:name/values endpoint. #15399
breaking[CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed. #15376
breaking[CHANGE] Set the GOMEMLIMIT variable automatically to match the Linux container memory limit. Use --no-auto-gomemlimit to disable it. The auto-gomemlimit feature flag was removed. #15373
breaking[CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a fallback_scrape_protocol in the scrape config. #15136
other[CHANGE] Remote-write: default enable_http2 to false. #15219
other[CHANGE] Scraping: normalize "le" and "quantile" label values upon ingestion. #15164
other[CHANGE] Scraping: config scrape_classic_histograms was renamed to always_scrape_classic_histograms. #15178
breaking[CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657
other[CHANGE] Disallow configuring AM with the v1 api. #13883
other[CHANGE] regexp . now matches all characters (performance improvement). #14505
other[CHANGE] holt_winters is now called double_exponential_smoothing and moves behind the experimental-promql-functions feature flag. #14930
other[CHANGE] API: The OTLP receiver endpoint can now be enabled using --web.enable-otlp-receiver instead of --enable-feature=otlp-write-receiver. #14894
feature[CHANGE] Prometheus will not add or remove port numbers from the target address. no-default-scrape-port feature flag removed. #14160
other[CHANGE] Logging: the format of log lines has changed a little, along with the adoption of Go's Structured Logging package. #14906
other[CHANGE] Don't create extra _created timeseries if feature-flag created-timestamp-zero-ingestion is enabled. #14738
other[CHANGE] Float literals and time durations being the same is now a stable fetaure. #15111
feature[CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch…
breaking[CHANGE] Kubernetes SD: Remove support for discovery.k8s.io/v1beta1 API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365
breaking[CHANGE] Kubernetes SD: Remove support for networking.k8s.io/v1beta1 API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365
breaking[CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed. #14705
breaking[CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807
breaking[CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed.…
breaking[CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead. #14747
deprecated[CHANGE] Remove deprecated remote-write-receiver,promql-at-modifier, and promql-negative-offset feature flags. #13456, #14526
deprecated[CHANGE] Remove deprecated storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention flags. #14640, #14643
feature[ENHANCEMENT] Tools: add debug printouts to promtool rules unit testing #15196
feature[ENHANCEMENT] PromQL: Introduce exponential interpolation for native histograms. #14677
feature[ENHANCEMENT] TSDB: Add support for ingestion of out-of-order native histogram samples. #14850, #14546
breaking[ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers. #13909
fix[BUGFIX] UI: fix selector / series formatting for empty metric names. #15341
fix[BUGFIX] PromQL: Fix stddev+stdvar aggregations to always ignore native histograms. #14941
fix[BUGFIX] PromQL: Fix stddev+stdvar aggregations to treat Infinity consistently. #14941
fix[BUGFIX] OTLP receiver: Preserve colons when generating metric names in suffix adding mode (this mode is always enabled, unless one uses Prometheus as a library). #15251
fix[BUGFIX] Scraping: Unit was missing when using protobuf format. #15095
fix[BUGFIX] PromQL: Only return "possible non-counter" annotation when rate returns points. #14910
fix[BUGFIX] TSDB: Chunks could have one unnecessary zero byte at the end. #14854
fix[BUGFIX] "superfluous response.WriteHeader call" messages in log. #14884
fix[BUGFIX] PromQL: Unary negation of native histograms. #14821
fix[BUGFIX] PromQL: Handle stale marker in native histogram series (e.g. if series goes away and comes back). #15025
fix[BUGFIX] Autoreload: Reload invalid yaml files. #14947
fix[BUGFIX] Scrape: Do not override target parameter labels with config params. #11029
+14 more items in the release notesfix[BUGFIX] round() function did not remove name label. #15250
feature[FEATURE] PromQL: Add experimental info function. #14495
other[FEATURE] Support UTF-8 characters in label names - feature flag utf8-names. #14482, #14880, #14736, #14727
feature[FEATURE] Scraping: Add the ability to set custom http_headers in config. #14817
other[FEATURE] Scraping: Support feature flag created-timestamp-zero-ingestion in OpenMetrics. #14356, #14815
other[FEATURE] Scraping: scrape_failure_log_file option to log failures to a file. #14734
other[FEATURE] OTLP receiver: Optional promotion of resource attributes to series labels. #14200
other[FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
feature[FEATURE] Promtool: tsdb create-blocks new option to add labels. #14403
feature[FEATURE] Promtool: promtool test adds --junit flag to format results. #14506
feature[FEATURE] TSDB: Add delayed-compaction feature flag, for people running many Prometheus to randomize timing. #12532
other[ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
other[ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
other[ENHANCEMENT] Remote Read client: Enable streaming remote read if the server supports it. #11379
other[ENHANCEMENT] Remote-Write: Don't reshard if we haven't successfully sent a sample since last update. #14450
other[ENHANCEMENT] PromQL: Delay deletion of name label to the end of the query evaluation. This is experimental and enabled under the feature-flag promql-delayed-name-removal. #14477
other[ENHANCEMENT] PromQL: Experimental sort_by_label and sort_by_label_desc sort by all labels when label is equal. #14655, #14985
other[ENHANCEMENT] PromQL: Clarify error message logged when Go runtime panic occurs during query evaluation. #14621
other[ENHANCEMENT] PromQL: Use Kahan summation for better accuracy in avg and avg_over_time. #14413
other[ENHANCEMENT] Tracing: Improve PromQL tracing, including showing the operation performed for aggregates, operators, and calls. #14816
other[ENHANCEMENT] API: Support multiple listening addresses. #14665
other[ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
other[PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948, #15120
other[PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
fix[BUGFIX] PromQL: make sort_by_label stable. #14985
fix[BUGFIX] SD: Fix dropping targets (with feature flag new-service-discovery-manager). #13147
fix[BUGFIX] SD: Stop storing stale targets (with feature flag new-service-discovery-manager). #13622
fix[BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
fix[BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
fix[BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
fix[BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14513, #14575, #14598, #14609, #14611, #14771, #14821
fix[BUGFIX] TSDB: Only count unknown record types in record_decode_failures_total metric. #14042
fix[BUGFIX] Scraping: allow multiple samples on same series, with explicit timestamps. #14685
fix[BUGFIX] Docker SD: fix crash in match_first_network mode when container is reconnected to a new network. #14654
fix[BUGFIX] PromQL: fix experimental native histogram counter reset detection on stale samples. #14514
fix[BUGFIX] PromQL: fix experimental native histograms getting corrupted due to vector selector bug in range queries. #14538
fix[BUGFIX] PromQL: fix experimental native histogram memory corruption when using histogram_count or histogram_sum. #14605
other[CHANGE] Remote-Write: highest_timestamp_in_seconds and queue_highest_sent_timestamp_seconds metrics now initialized to 0. #14437
other[CHANGE] API: Split warnings from info annotations in API response. #14327
other[FEATURE] Remote-Write: Version 2.0 experimental, plus metadata in WAL via feature flag metadata-wal-records (defaults on). #14395,#14427,#14444
feature[FEATURE] PromQL: add limitk() and limit_ratio() aggregation operators. #12503
other[ENHANCEMENT] PromQL: Accept underscores in literal numbers, e.g. 1_000_000 for 1 million. #12821
other[ENHANCEMENT] PromQL: float literal numbers and durations are now interchangeable (experimental). Example: time() - my_timestamp > 10m. #9138
other[ENHANCEMENT] PromQL: use Kahan summation for sum(). #14074,#14362
other[ENHANCEMENT] PromQL (experimental native histograms): Optimize histogram_count and histogram_sum functions. #14097
feature[ENHANCEMENT] TSDB: Better support for out-of-order experimental native histogram samples. #14438
other[ENHANCEMENT] TSDB: Optimise seek within index. #14393
other[ENHANCEMENT] TSDB: Optimise deletion of stale series. #14307
other[ENHANCEMENT] TSDB: Reduce locking to optimise adding and removing series. #13286,#14286
other[ENHANCEMENT] TSDB: Small optimisation: streamline special handling for out-of-order data. #14396,#14584
other[ENHANCEMENT] Regexps: Optimize patterns with multiple prefixes. #13843,#14368
other[ENHANCEMENT] Regexps: Optimize patterns containing multiple literal strings. #14173
other[ENHANCEMENT] AWS SD: expose Primary IPv6 addresses as __meta_ec2_primary_ipv6_addresses. #14156
feature[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks. #10490
other[ENHANCEMENT] OpenStack SD: Use flavor.original_name if available. #14312
other[ENHANCEMENT] UI (experimental native histograms): more accurate representation. #13680,#14430
other[ENHANCEMENT] Agent: out_of_order_time_window config option now applies to agent. #14094
other[ENHANCEMENT] Notifier: Send any outstanding Alertmanager notifications when shutting down. #14290
feature[ENHANCEMENT] Rules: Add label-matcher support to Rules API. #10194
feature[ENHANCEMENT] HTTP API: Add url to message logged on error while sending response. #14209
fix[BUGFIX] CLI: escape | characters when generating docs. #14420
fix[BUGFIX] PromQL (experimental native histograms): Fix some binary operators between native histogram values. #14454
fix[BUGFIX] TSDB: LabelNames API could fail during compaction. #14279
fix[BUGFIX] TSDB: Fix rare issue where pending OOO read can be left dangling if creating querier fails. #14341
fix[BUGFIX] TSDB: fix check for context cancellation in LabelNamesFor. #14302
fix[BUGFIX] Rules: Fix rare panic on reload. #14366
fix[BUGFIX] Config: In YAML marshalling, do not output a regexp field if it was never set. #14004
fix[BUGFIX] Remote-Write: reject samples with future timestamps. #14304
fix[BUGFIX] Remote-Write: Fix data corruption in remote write if max_sample_age is applied. #14078
fix[BUGFIX] Notifier: Fix Alertmanager discovery not updating under heavy load. #14174
fix[BUGFIX] Regexes: some Unicode characters were not matched by case-insensitive comparison. #14170,#14299
fix[BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515
feature[ENHANCEMENT] TSDB: Add backward compatibility with the upcoming TSDB block index v3 #16762
fix[BUGFIX] Top-level: Update GOGC before loading TSDB #16521
fix[BUGFIX] Runtime: fix GOGC is being set to 0 when installed with empty prometheus.yml file resulting high cpu usage. #16090
fix[BUGFIX] Scrape: fix dropping valid metrics after previous scrape failed. #16220
fix[BUGFIX] Scraping: allow multiple samples on same series, with explicit timestamps. #14685, #14740
fixFix a bug where Prometheus would crash with a segmentation fault if a remote-read request accessed a block on disk at about the same time as TSDB created a new block.
fix[BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515,#14523
fix[BUGFIX] Remote-write: stop dropping samples in catch-up #14446
other[CHANGE] Rules: Execute 1 query instead of N (where N is the number of alerts within alert rule) when restoring alerts. #13980 #14048
other[CHANGE] Runtime: Change GOGC threshold from 100 to 75 #14176 #14285
feature[FEATURE] Rules: Add new option query_offset for each rule group via rule group configuration file and rule_query_offset as part of the global configuration to have more resilience for remote write delays. #14061 #14216 #14273
feature[ENHANCEMENT] Rules: Add rule_group_last_restore_duration_seconds metric to measure the time it takes to restore a rule group. #13974
other[ENHANCEMENT] OTLP: Improve remote write format translation performance by using label set hashes for metric identifiers instead of string based ones. #14006 #13991
other[ENHANCEMENT] TSDB: Optimize querying with regexp matchers. #13620
fix[BUGFIX] OTLP: Don't generate target_info unless there are metrics and at least one identifying label is defined. #13991
fix[BUGFIX] Scrape: Do no try to ingest native histograms when the native histograms feature is turned off. This happened when protobuf scrape was enabled by for example the created time feature. #13987
fix[BUGFIX] Scaleway SD: Use the instance's public IP if no private IP is available as the address meta label. #13941
fix[BUGFIX] Query logger: Do not leak file descriptors on error. #13948
fix[BUGFIX] TSDB: Let queries with heavy regex matches be cancelled and not use up the CPU. #14096 #14103 #14118 #14199
fix[BUGFIX] API: Do not warn if result count is equal to the limit, only when exceeding the limit for the series, label-names and label-values APIs. #14116
fix[BUGFIX] TSDB: Fix head stats and hooks when replaying a corrupted snapshot. #14079
fix[CHANGE] TSDB: Fix the predicate checking for blocks which are beyond the retention period to include the ones right at the retention boundary. #9633
feature[FEATURE] Kubernetes SD: Add a new metric prometheus_sd_kubernetes_failures_total to track failed requests to Kubernetes API. #13554
feature[FEATURE] Kubernetes SD: Add node and zone metadata labels when using the endpointslice role. #13935
feature[FEATURE] Azure SD/Remote Write: Allow usage of Azure authorization SDK. #13099
other[FEATURE] Alerting: Support native histogram templating. #13731
other[FEATURE] Linode SD: Support IPv6 range discovery and region filtering. #13774
other[ENHANCEMENT] PromQL: Performance improvements for queries with regex matchers. #13461
other[ENHANCEMENT] PromQL: Performance improvements when using aggregation operators. #13744
other[ENHANCEMENT] PromQL: Validate label_join destination label. #13803
other[ENHANCEMENT] Scrape: Increment prometheus_target_scrapes_sample_duplicate_timestamp_total metric on duplicated series during one scrape. #12933
other[ENHANCEMENT] TSDB: Many improvements in performance. #13742 #13673 #13782
other[ENHANCEMENT] TSDB: Pause regular block compactions if the head needs to be compacted (prioritize head as it increases memory consumption). #13754
other[ENHANCEMENT] Observability: Improved logging during signal handling termination. #13772
other[ENHANCEMENT] Observability: All log lines for drop series use "num_dropped" key consistently. #13823
other[ENHANCEMENT] Observability: Log chunk snapshot and mmaped chunk replay duration during WAL replay. #13838
other[ENHANCEMENT] Observability: Log if the block is being created from WBL during compaction. #13846
fix[BUGFIX] PromQL: Fix inaccurate sample number statistic when querying histograms. #13667
fix[BUGFIX] PromQL: Fix histogram_stddev and histogram_stdvar for cases where the histogram has negative buckets. #13852
fix[BUGFIX] PromQL: Fix possible duplicated label name and values in a metric result for specific queries. #13845
fix[BUGFIX] Scrape: Fix setting native histogram schema factor during scrape. #13846
fix[BUGFIX] TSDB: Fix counting of histogram samples when creating WAL checkpoint stats. #13776
fix[BUGFIX] TSDB: Fix cases of compacting empty heads. #13755
fix[BUGFIX] TSDB: Count float histograms in WAL checkpoint. #13844
fix[BUGFIX] Remote Read: Fix memory leak due to broken requests. #13777
fix[BUGFIX] API: Stop building response for /api/v1/series/ when the API request was cancelled. #13766
fix[BUGFIX] promtool: Fix panic on promtool tsdb analyze --extended when no native histograms are present. #13976
fixBugfix release.
fix[BUGFIX] Notifier: could hang when using relabeling on alerts #13861