Major Improvements
-
Partitioned Execution Support:
Added support for splitting jobs across multiple executions with automatic partition management. The feature includes:- Partition assignment and tracking
- Independent execution progress monitoring
- Granular failure handling with retry of only failed partitions
- Each execution receives its partition details through environment variables, enabling partition-aware processing when needed
-
S3 Input Partitioning:
Added automatic data distribution for S3 inputs across multiple executions using configurable strategies:- Multiple partitioning strategies: Users can choose between object-based distribution for even splitting, regex patterns for structured data, substring matching for fixed formats, or date-based partitioning for temporal data
- Even distribution of data without requiring custom partition code
- Support for shared data access through non-partitioned inputs
- Automatic data subset assignment to each execution
Full Changelog: v1.6.0...v1.6.1