Logstash 9.3.6 → 9.5.5
17 releases · 5 breaking changes · 0 security fixes · 0 advisories fixed
0security5breaking2deprecated27feature41fix
Breaking changes
- breaking Drop support for Logstash 7.x by requiring elasticsearch gem >= 8. Logstash 8+ continues to work as before #213 in 9.5.0
- breaking Drop support for Logstash 7.x by requiring elasticsearch gem >= 8. Logstash 8+ continues to work as before #252 in 9.5.0
- breaking Remove explicit lz4-java dependency (now transitive from Kafka client) in 9.5.0
- breaking Breaking: partitioner options default and uniform_sticky are removed in 9.5.0
- breaking Remove duplicated deprecation log entry #208 in 9.5.0
Security changes
None in the release notes.
Deprecations
- deprecated Replace deprecated Aws::S3::Object#upload_file in favor of Aws::S3::TransferManager#upload_file #67 in 9.4.0
- deprecated Replace deprecated File.exists? with File.exist? for Ruby 3.4 (JRuby 10) compatibility #65 in 9.4.0
Every release in between
otherUpdate bundled JDK to 21.0.12.1 build 1 #19510
otherUpdate bundled JRuby to 10.0.7.0 #19608
otherUpdate Jackson to 2.21.7 and jrjackson to 0.5.4 #19442 #19577
otherUpgrade rubyzip dependency to 3.x #19501
otherUpgrade zstd-jni dependency to 1.5.7-14 #19512
otherDocument that the lumberjack input plugin should not be used for Logstash-to-Logstash communication #19505
otherUpdate Netty dependency to 4.1.138.Final #579
otherUpdate Netty dependency to 4.1.138.Final #241
otherUpdate Netty dependency to 4.1.138.Final #290
otherUpgrade march_hare to consume 4.10+ #87
fixFix for processing the HA Proxy protocol, considering the line can be split in multiple buffers. #286
otherUpgrade zstd-jni dependency #279
fixFix: terminate idle scheduler work threads on plugin close, which leaked one thread per pipeline reload #7
featureInclude timestamp in health report snapshot API response #19422
otherRequire httpclient5 version 5.6.4 or later #501
otherUpdate jackson dependency to 3.1.6 #499
otherUpdate proton-j to 0.35.0 #122
otherUpdate jackson dependency to 2.21.6 #121
otherUpdate Netty dependency to 4.1.137.Final #576
otherUpdate Netty dependency to 4.1.137.Final #239
featureAdd optional HTTP traffic logging (disabled by default). To enable it, turn on debug logging. #237
otherUpdate Netty dependency to 4.1.137.Final #284
fixFix access method for large tables to avoid materializing the full result set in memory and triggering an out-of-memory error. #204
otherPrevent concurrent Sequel JDBC subadapter initialization races by preloading adapter at driver load time #203
otherRequire httpclient5 version 5.6.4 or later #277
otherUpgrades kafka-avro-serializer dependency to v8.3.1, and constrains transitive dependency on Apache httpclient5 to be at least v5.6.3 #273
otherUpdate jackson dependency to 2.21.6 #274
otherRequire mail dependency 2.9 or later #72
fixMade BufferedTokenizer thread safe and usable in multithreaded contexts; its flush operation now returns an error when the remainder token overruns the size limit, and logs when data is dropped #19345 #19312
fixHandle a NoSuchFileException that could occur while resolving the oldest dead letter queue segment file #19409
otherUpdate concurrent-ruby to 1.3.8 #19428
fixFix NameError: uninitialized constant Bignum that prevented the codec from loading on Logstash 9.4+ #8
otherThe edn gem references the Bignum constant, which Ruby removed in 3.2 (shipped by the JRuby in Logstash 9.4+). This raised a NameError when the codec registered, so any pipeline using the edn codec failed to start. Alias the removed Fixnum/Bignum constants to Integer before requiring edn.
otherThe edn gem references the Bignum constant, which Ruby removed in 3.2 (shipped by the JRuby in Logstash 9.4+). This raised a NameError when the codec registered, so any pipeline using the edn_lines codec failed to start. Alias the removed Fixnum/Bignum constants to Integer before requiring edn.
fixFix NameError: uninitialized constant Fixnum when using the MURMUR3 algorithm on Logstash 9.4+ (Ruby 3.2+, which removed Fixnum in favor of Integer) #19
fixFixes an issue where failing to load a dictionary could cause the plugin to continue to run with a missing or partially-updated dictionary; this issue was especially noticeable when configured with refresh_behaviour => replace, which clears the dictionary before loading the replacement #112.
fixFix NameError: uninitialized constant Fixnum when reading a JMS MapMessage on Logstash 9.4+ #63
otherThe jruby-jms gem references the Fixnum constant, which Ruby removed in 3.2 (shipped by the JRuby in Logstash 9.4+). This raised a NameError while decoding a JMS MapMessage. Alias the removed Fixnum/Bignum constants to Integer before requiring jms.
fixUpdates common functionality used to dice data used in some codec plugins to validate oversized segments also on last part. Added logging when data is dropped because passed the size limit #19312
featureSend Logstash Metrics via OTLP #18857
featureAuto-reload pipelines on TLS certificate rotation #18978
featureExtend TLS auto-reload to CPM and monitoring service clients #19045
featureAdd support for cgroupv2 #18708
featureNew dead_letter_queue.flush_check_interval setting to control how frequently stale DLQ segment files are flushed #19036
featureRemoved legacy output concurrency mode. The deprecated concurrency :legacy option is no longer supported; outputs should use concurrency :single or concurrency :shared #19003
featureRemoved the pipe output plugin from the default plugin bundle #19159
fixPipeline metrics are now cleared when a pipeline fails to start, preventing stale metrics from being reported by the monitoring API #19091
fixFailures in plugin do_close no longer halt the pipeline shutdown sequence #19035
fixFixed forced-shutdown (double SIGINT) behavior for JRuby 10 #19017
fixAdded missed standard types during batch size estimation #19158
otherBump OpenTelemetry to 1.62.0 #19209
otherUpdate Netty dependency to 4.1.136.Final #575
otherSet SO_TIMEOUT on IOReactor to ensure NIO connections have a baseline socket timeout, preventing indefinite hangs when connection pool restore fails to apply the request-level timeout #480
otherSync up with Elasticsearch 9.5 branch to pull latest dependencies #476
fixFixes an issue where a field set by an integration pipeline to java.util.Date value-object representing a timestamp could not be converted to a timestamp #460
otherApplies Elasticsearch geoip module relocation changes #445
otherSupport Elastic Cloud API keys in the api_key option, which now accepts an id:api_key pair, its base64-encoded form, or an essu_ Cloud API key, and rejects an unrecognized format at startup #215
breakingDrop support for Logstash 7.x by requiring elasticsearch gem >= 8. Logstash 8+ continues to work as before #213
otherSupport Elastic Cloud API keys in the api_key option, which now accepts an id:api_key pair, its base64-encoded form, or an essu_ Cloud API key, and rejects an unrecognized format at startup #274
fixFix serverless request failure caused by conflicting compatible-with and Elastic-Api-Version headers when using elasticsearch-ruby v9 #269
breakingDrop support for Logstash 7.x by requiring elasticsearch gem >= 8. Logstash 8+ continues to work as before #252
fixFix to use the Content-type declared charset to decode the request body #230
otherUpdate Netty dependency to 4.1.136.Final #228
otherUpdate Netty dependency to 4.1.136.Final #281
fixFix connection leak on statement retry by opening JDBC connection once outside retry loop #201
fixFix sasl_jaas_config output configuration #245
otherUpdate Kafka client to 4.2.0 #243
breakingRemove explicit lz4-java dependency (now transitive from Kafka client)
otherDocument by_duration offset reset strategy (available since Apache Kafka 4.0.0)
otherMask sensitive sasl_jaas_config values in debug logs to prevent credential exposure #232
otherUpdate Kafka client to 4.1.0 and transitive dependencies #205
breakingBreaking: partitioner options default and uniform_sticky are removed
otherlinger_ms default value changed from 0 to 5
featureAdd group_protocols option for configuring Kafka consumer rebalance protocol
featureSetting group_protocol => consumer opts in to the new consumer group protocol
breakingRemove duplicated deprecation log entry #208
otherGenerate error events with _snmpfailure tag when all SNMP operations fail and the response data is empty (timeout for example) #92
featureHandle partial responses and errors gracefully: add tag_on_failure (default: ["_snmpfailure"]) to tag events when SNMP operations fail, and allow_partial_response (default: false) to preserve partial data from failed walk/table operations #91
featureAdded support for IPv6 addresses #16
otherUpdate bundled JDK to 21.0.12.1 build 1 #19508
otherUpdate bundled JRuby to 10.0.7.0 #19609
otherUpdate Jackson to 2.21.7 and jrjackson to 0.5.4 #19442 #19577
otherUpgrade rubyzip dependency to 3.x #19501
otherUpgrade zstd-jni dependency to 1.5.7-14 #19512
otherDocument that the lumberjack input plugin should not be used for Logstash-to-Logstash communication #19505
otherUpdate Netty dependency to 4.1.138.Final #579
otherUpdate Netty dependency to 4.1.138.Final #241
otherUpdate Netty dependency to 4.1.138.Final #290
otherUpgrade march_hare to consume 4.10+ #87
otherRequire httpclient5 5.6.4 or later #502
otherUpdate jackson dependency to 3.1.6 #498
otherUpdate proton-j to 0.35.0 #122
fixFix for processing the HA Proxy protocol, considering the line can be split in multiple buffers. #286
fixFix access method for large tables to avoid materializing the full result set in memory and triggering an out-of-memory error. #204
otherPrevent concurrent Sequel JDBC subadapter initialization races by preloading adapter at driver load time #203
otherUpgrade zstd-jni dependency #280
fixFix: terminate idle scheduler work threads on plugin close, which leaked one thread per pipeline reload #7
otherRequiere mail dependency 2.9 or later #72
featureInclude timestamp in health report snapshot API response #19422
fixFix a NoSuchFileException in the dead letter queue that could occur during oldest segment resolution when a segment file is concurrently removed #19409
otherUpdated concurrent-ruby gem #19428
fixFix NameError: uninitialized constant Bignum that prevented the codec from loading on Logstash 9.4+ #8
otherThe edn gem references the Bignum constant, which Ruby removed in 3.2 (shipped by the JRuby in Logstash 9.4+). This raised a NameError when the codec registered, so any pipeline using the edn codec failed to start. Alias the removed Fixnum/Bignum constants to Integer before requiring edn.
otherThe edn gem references the Bignum constant, which Ruby removed in 3.2 (shipped by the JRuby in Logstash 9.4+). This raised a NameError when the codec registered, so any pipeline using the edn_lines codec failed to start. Alias the removed Fixnum/Bignum constants to Integer before requiring edn.
fixFix NameError: uninitialized constant Fixnum when using the MURMUR3 algorithm on Logstash 9.4+ (Ruby 3.2+, which removed Fixnum in favor of Integer) #19
otherUpdate jackson dependency to 2.21.6 #121
otherUpdate Netty dependency to 4.1.137.Final #576
otherUpdate Netty dependency to 4.1.137.Final #239
featureAdd optional HTTP traffic logging (disabled by default). To enable it, turn on debug logging. #237
fixFix NameError: uninitialized constant Fixnum when reading a JMS MapMessage on Logstash 9.4+ #63
otherThe jruby-jms gem references the Fixnum constant, which Ruby removed in 3.2 (shipped by the JRuby in Logstash 9.4+). This raised a NameError while decoding a JMS MapMessage. Alias the removed Fixnum/Bignum constants to Integer before requiring jms.
otherUpdate Netty dependency to 4.1.137.Final #284
otherUpdate jackson dependency to 2.21.6 #275
otherUpdated JDK to 21.0.12 build 8 #19365
otherApply the default 30-second socket timeout to low-level HTTP connections so stalled connections time out consistently. #480
fixFixes an issue where failing to load a dictionary could cause the plugin to continue to run with a missing or partially-updated dictionary; this issue was especially noticeable when configured with refresh_behaviour => replace, which clears the dictionary before loading the replacement #112.
otherUpdated JRuby to 10.0.6.0 #19279
otherUpdated Jackson and jrjackson dependencies #19286
otherUpgraded Puma to 8.x #19234
otherSync up with Elasticsearch 9.4 branch to pull latest dependencies #477
otherUpdate jackson dependency to 3.1.4 #467
otherUpdate jackson dependency to 2.21.4 #118
fixWhen configured to use a port that is already in use, the failure is now propagated to the pipeline #537 This fixes an issue where a misconfigured input could retry indefinitely while Logstash's health report continued to report the pipeline as healthy.
fixWhen configured to use a port that is already in use, the failure is now propagated to the pipeline #221 This fixes an issue where a misconfigured input could retry indefinitely while Logstash's health report continued to report the pipeline as healthy.
otherUpdate jackson dependency to 2.21.4 #269
fixFix serverless compatibility: nil params in pool requests and unsupported template settings #1276
otherSupport Elastic Cloud API keys in the api_key option, which now accepts an id:api_key pair, its base64-encoded form, or an essu_ Cloud API key, and rejects an unrecognized format at startup #1274
feature[Doc] Add note for index option #1269
featureBatch size estimation improvements #19180
otherLogstash's file descriptors usage, limits and sizing guidance documented #19206
otherUpgrades jackson.core to 2.21.2 and nimbus-jose-jwt to 10.9 versions. #117
feature[DOCS] Added recommendation to migrate to the Kafka integration plugin #116
otherUpdate Netty dependency to 4.1.135.Final #567
otherUpdate Netty dependency to 4.1.135.Final #219
otherUpdate Netty dependency to 4.1.135.Final #274
fixFixes a metric leak in _node/stats when a pipeline repeatedly fails to start with config.reload.automatic: true. Previously, each retry left a fresh set of plugin metric entries in the collector, causing the stats payload to grow indefinitely #19120
otherUpdated jruby-openssl to 0.16.0 #19116
otherUpgraded jrjackson and fasterxml.jackson #19126
fixFixes an issue where a field set by an integration pipeline to java.util.Date value-object representing a timestamp could not be converted to a timestamp #462
otherInclude httpclient5/httpcore5 from the elasticsearch-java artifact #458
otherUpgrades jackson.core to 2.21.2 version #255
otherUpdate Netty dependency to 4.1.134.Final #541
otherUpdate Netty dependency to 4.1.134.Final #217
otherUpdate Netty dependency to 4.1.134.Final #257
other[DOC] Document command string form limitation #6
otherUpdate bundled JDK to 21.0.11 build 10.
otherUpdate Netty dependency to 4.1.133.Final #539
otherUpdate Netty dependency to 4.1.133.Final #216
otherUpdate Netty dependency to 4.1.133.Final #256
fixWhen configured to use a port that is already in use, the failure is now propagated to the pipeline. This fixes an issue where a misconfigured input could retry indefinitely while Logstash's health report continued to report the pipeline as healthy #250
otherImproves the logging experience when DLQ used #1253.
featureauto: enables pipeline crash recovery for pipelines backed by the persistent queue
featurefalse (default): do not automate recovery of crashed pipelines
featuretrue: enables pipeline crash recovery for any pipeline, even if backed by the ephemeral memory queue (risk: data loss)
featurePerformance improvements which saves ~40% CPU resource on DLQ segment file lookup operations 19013
otherUpdate JRuby to 10.0.5.0
otherUpdate bundled JDK to 21.0.10 build 7
featureFeature: Add a warning log message when the number of tasks stored in memory exceeds the configured threshold #125
fixUse milliseconds timestamp precision in S3 input to fix the skip backup and delete object issue in S3-compatible storage services #60
deprecatedReplace deprecated Aws::S3::Object#upload_file in favor of Aws::S3::TransferManager#upload_file #67
deprecatedReplace deprecated File.exists? with File.exist? for Ruby 3.4 (JRuby 10) compatibility #65
otherRe-packaging the plugin #63
featureAdd cutoff_second configuration option to S3 input plugin #59
featureAdd precision setting to support nanosecond precision timestamps #165
otherms (default): timestamps are stored with millisecond precision
otherit keeps the same behavior as before for backward compatibility
otherfractional seconds are truncated to 3 digits
othercustom parsing formats use joda-time library
otherns: timestamps are stored with nanosecond precision
otherfractional seconds support up to 9 digits
othercustom parsing formats use java.time
otherISO8601 now accepts up to 9 fractional-second digits
otherApply an 'error' tag to any event that fails the de-dotting process #26
featureAdd JRuby 10 support: replace removed NativeException with RaiseException, source JRuby from Logstash vendor directory instead of pinning Maven version #96
otherInclude Elasticsearch web-utils JAR into the plugin to keep registered_domain processor dependencies #397
fixFixed set_security_user processor to behave consistently with other unsupported processors (inference, enrich) by tagging events with _ingest_pipeline_failure #269
fixFix fingerprint instability for Hash and Array field values caused by JRuby 10 changing Hash#inspect formatting #79
fixFix a regression introduced in 11.8.7 where sasl_jaas_config was incorrectly typed in the output plugin, crashing Logstash on startup when a SASL Kafka output configuration was present #247
fixAdd JRuby 10 support: fix integer conversion precision loss beyond 2^53 caused by parse_signed_hex_str routing all strings through Float() #178
otherUpdates the gelf dependency #77
otherUpdated JRuby to 9.4.15.0 #19290
otherUpdated Jackson and jrjackson dependencies #19289
otherSync up with Elasticsearch 9.3 branch to pull latest dependencies #478
otherUpdate jackson dependency to 3.1.4 #470
otherUpdate jackson dependency to 2.21.4 #118
fixWhen configured to use a port that is already in use, the failure is now propagated to the pipeline #537 This fixes an issue where a misconfigured input could retry indefinitely while Logstash's health report continued to report the pipeline as healthy.
fixWhen configured to use a port that is already in use, the failure is now propagated to the pipeline #221 This fixes an issue where a misconfigured input could retry indefinitely while Logstash's health report continued to report the pipeline as healthy.
otherUpdate jackson dependency to 2.21.4 #269
fixFix serverless compatibility: nil params in pool requests and unsupported template settings #1276
otherSupport Elastic Cloud API keys in the api_key option, which now accepts an id:api_key pair, its base64-encoded form, or an essu_ Cloud API key, and rejects an unrecognized format at startup #1274
feature[Doc] Add note for index option #1269
otherUpgrade puma to 8.x #19228
otherNo user-facing changes in Logstash plugins.