- [BREAKING-EXPERIMENTAL] The
batch_dataattribute ofBatchRequesthas been removed. To pass in in-memory dataframes at runtime, the newRuntimeDataConnectorshould be used - [BREAKING-EXPERIMENTAL]
RuntimeDataConnectormust now be passed Batch Requests of typeRuntimeBatchRequest - [BREAKING-EXPERIMENTAL] The
PartitionDefinitionSubsetclass has been removed - the parent classIDDictis used in its place - [BREAKING-EXPERIMENTAL]
partition_requestwas renameddata_connector_query. The relatedPartitionRequestclass has been removed - the parent classIDDictis used in its place - [BREAKING-EXPERIMENTAL]
partition_definitionwas renamedbatch_identifiers. The relatedPartitionDefinitionclass has been removed - the parent classIDDictis used in its place - [BREAKING-EXPERIMENTAL] The
PartitionQueryclass has been renamed toBatchFilter - [BREAKING-EXPERIMENTAL] The
batch_identifierskey onDataConnectorQuery(formerlyPartitionRequest) has been changed tobatch_filter_parameters - [ENHANCEMENT] Added a new
RuntimeBatchRequestclass, which can be used alongsideRuntimeDataConnectorto specify batches at runtime with either an in-memory dataframe, path (filesystem or s3), or sql query - [ENHANCEMENT] Added a new
RuntimeQueryBatchSpecclass - [BUGFIX] Fixed issue where Sorters were not being applied correctly when
data_connector_querycontained limit or index #2617 - [DOCS] Updated docs to reflect above class name changes
- [DOCS] Added the following docs: "How to configure sorting in Data Connectors", "How to configure a Runtime Data Connector", "How to create a Batch Request using an Active Data Connector", "How to load a database table, view, or query result as a Batch"
- [DOCS] Updated the V3 API section of the following docs: "How to load a Pandas DataFrame as a Batch", "How to load a Spark DataFrame as a Batch"