For installation instructions check out the getting started guide.
This is a major version release, for more information and guidance on how to migrate please refer to https://benthos.dev/docs/guides/migration/v4.
Added
- In Bloblang it is now possible to reference the
root
of the document being created within a mapping query. - The
nats_jetstream
input now supports pull consumers. - Field
max_number_of_messages
added to theaws_sqs
input. - Field
file_output_path
added to theprometheus
metrics type. - Unit test definitions can now specify a
label
as atarget_processors
value. - New connection settings for all sql components.
- New experimental
snowflake_put
output. - New experimental
gcp_cloud_storage
cache. - Field
regexp_topics
added to thekafka_franz
input. - The
hdfs
outputdirectory
field now supports interpolation functions. - The cli
list
subcommand now supports acue
format. - Field
jwt.headers
added to all HTTP client components. - Output condition
file_json_equals
added to config unit test definitions.
Fixed
- The
sftp
output no longer opens files in both read and write mode. - The
aws_sqs
input withreset_visibility
set tofalse
will no longer reset timeouts on pending messages during gracefully shutdown. - The
schema_registry_decode
processor now handles AVRO logical types correctly. Details in #1198 and #1161 and also in linkedin/goavro#242.
Changed
- All components, features and configuration fields that were marked as deprecated have been removed.
- The
pulsar
input and output are no longer included in the default Benthos builds. - The field
pipeline.threads
field now defaults to-1
, which automatically matches the host machine CPU count. - Old style interpolation functions (
${!json:foo,1}
) are removed in favour of the newer Bloblang syntax (${! json("foo") }
). - The Bloblang functions
meta
,root_meta
,error
andenv
now returnnull
when the target value does not exist. - The
clickhouse
SQL driver Data Source Name format parameters have been changed due to a client library update. This also means placeholders insql_raw
components should use dollar syntax. - Docker images no longer come with a default config that contains generated environment variables, use
-s
flag arguments instead. - All cache components have had their retry/backoff fields modified for consistency.
- All cache components that support a general default TTL now have a field
default_ttl
with a duration string, replacing the previous field. - The
http
processor andhttp_client
output now execute message batch requests as individual requests by default. This behaviour can be disabled by explicitly settingbatch_as_multipart
totrue
. - Outputs that traditionally wrote empty newlines at the end of batches with >1 message when using the
lines
codec (socket
,stdout
,file
,sftp
) no longer do this by default. - The
switch
output fieldretry_until_success
now defaults tofalse
. - All AWS components now have a default
region
field that is empty, allowing environment variables or profile values to be used by default. - Serverless distributions of Benthos (AWS lambda, etc) have had the default output config changed to reject messages when the processing fails, this should make it easier to handle errors from invocation.
- The standard metrics emitted by Benthos have been largely simplified and improved, for more information check out the metrics page.
- The default metrics type is now
prometheus
. - The
http_server
metrics type has been renamed tojson_api
. - The
stdout
metrics type has been renamed tologger
. - The
logger
configuration section has been simplified, withlogfmt
being the new default format. - The
logger
fieldadd_timestamp
is nowfalse
by default. - Field
parts
has been removed from all processors. - Field
max_in_flight
has been removed from a range of output brokers as it no longer required. - The
dedupe
processor now acts upon individual messages by default, and thehash
field has been removed. - The
log
processor now executes for each individual message of a batch. - The
sleep
processor now executes for each individual message of a batch. - Go API: Module name has changed to
github.com/benthosdev/benthos/v4
. - Go API: All packages within the
lib
directory have been removed in favour of the newer APIs withinpublic
. - Go API: Distributed tracing is now via the Open Telemetry client library.
The full change log can be found here.