redpanda-data/connect v4.0.0

on GitHub

For installation instructions check out the getting started guide.

This is a major version release, for more information and guidance on how to migrate please refer to https://benthos.dev/docs/guides/migration/v4.

Added

• In Bloblang it is now possible to reference the root of the document being created within a mapping query.
• The nats_jetstream input now supports pull consumers.
• Field max_number_of_messages added to the aws_sqs input.
• Field file_output_path added to the prometheus metrics type.
• Unit test definitions can now specify a label as a target_processors value.
• New connection settings for all sql components.
• New experimental snowflake_put output.
• New experimental gcp_cloud_storage cache.
• Field regexp_topics added to the kafka_franz input.
• The hdfs output directory field now supports interpolation functions.
• The cli list subcommand now supports a cue format.
• Field jwt.headers added to all HTTP client components.
• Output condition file_json_equals added to config unit test definitions.

Fixed

• The sftp output no longer opens files in both read and write mode.
• The aws_sqs input with reset_visibility set to false will no longer reset timeouts on pending messages during gracefully shutdown.
• The schema_registry_decode processor now handles AVRO logical types correctly. Details in #1198 and #1161 and also in linkedin/goavro#242.

Changed

• All components, features and configuration fields that were marked as deprecated have been removed.
• The pulsar input and output are no longer included in the default Benthos builds.
• The field pipeline.threads field now defaults to -1, which automatically matches the host machine CPU count.
• Old style interpolation functions (${!json:foo,1}) are removed in favour of the newer Bloblang syntax (${! json("foo") }).
• The Bloblang functions meta, root_meta, error and env now return null when the target value does not exist.
• The clickhouse SQL driver Data Source Name format parameters have been changed due to a client library update. This also means placeholders in sql_raw components should use dollar syntax.
• Docker images no longer come with a default config that contains generated environment variables, use -s flag arguments instead.
• All cache components have had their retry/backoff fields modified for consistency.
• All cache components that support a general default TTL now have a field default_ttl with a duration string, replacing the previous field.
• The http processor and http_client output now execute message batch requests as individual requests by default. This behaviour can be disabled by explicitly setting batch_as_multipart to true.
• Outputs that traditionally wrote empty newlines at the end of batches with >1 message when using the lines codec (socket, stdout, file, sftp) no longer do this by default.
• The switch output field retry_until_success now defaults to false.
• All AWS components now have a default region field that is empty, allowing environment variables or profile values to be used by default.
• Serverless distributions of Benthos (AWS lambda, etc) have had the default output config changed to reject messages when the processing fails, this should make it easier to handle errors from invocation.
• The standard metrics emitted by Benthos have been largely simplified and improved, for more information check out the metrics page.
• The default metrics type is now prometheus.
• The http_server metrics type has been renamed to json_api.
• The stdout metrics type has been renamed to logger.
• The logger configuration section has been simplified, with logfmt being the new default format.
• The logger field add_timestamp is now false by default.
• Field parts has been removed from all processors.
• Field max_in_flight has been removed from a range of output brokers as it no longer required.
• The dedupe processor now acts upon individual messages by default, and the hash field has been removed.
• The log processor now executes for each individual message of a batch.
• The sleep processor now executes for each individual message of a batch.
• Go API: Module name has changed to github.com/benthosdev/benthos/v4.
• Go API: All packages within the lib directory have been removed in favour of the newer APIs within public.
• Go API: Distributed tracing is now via the Open Telemetry client library.

The full change log can be found here.