Added
LLMReranker
now supports custom prompts as well as custom response parsers allowing for other ranking scales apart from default 1-5.pw.io.kafka.write
andpw.io.nats.write
now supportColumnReference
as a topic name. When aColumnReference
is provided, each message's topic is determined by the corresponding column value.pw.io.python.write
acceptingConnectorObserver
as an alternative topw.io.subscribe
.pw.io.iceberg.read
andpw.io.iceberg.write
now support S3 as data backend and AWS Glue catalog implementations.- All output connectors now support the
sort_by
field for ordering output within a single minibatch. - A new UDF executor
pw.udfs.fully_async_executor
. It allows for creation of non-blocking asynchronous UDFs which results can be returned in the future processing time. - A Future data type to represent results of fully asynchronous UDFs.
pw.Table.await_futures
method to wait for results of fully asynchronous UDFs.pw.io.deltalake.write
now supports partition columns specification.
Changed
- BREAKING: Changed the interface of
LLMReranker
, theuse_logit_bias
,cache_strategy
,retry_strategy
andkwargs
arguments are no longer supported. - BREAKING: LLMReranker no longer inherits from pw.UDF
- BREAKING:
pw.stdlib.utils.AsyncTransformer.output_table
now returns a table with columns with Future data type. pw.io.deltalake.read
can now read append-only tables without requiring explicit specification of primary key fields.