slingdata-io/sling-cli v1.4.1

on GitHub

Sling v1.4.1 (2025-02-01T21:46:18Z)

Major Changes

1. Pipeline Implementation

   • Added new Pipeline type for executing sequential steps
   • Introduced pipeline configuration loading from YAML/JSON files
   • Added support for pipeline steps with various types (log, replication, command, etc.)

2. State Management Refactoring

   • Renamed Hooks to State in runtime state management
   • Created RuntimeState interface
   • Added separate ReplicationState and PipelineState implementations

3. Chunking Functionality

   • Added support for chunking data processing based on column ranges
   • Implemented ProcessChunks() method for handling data partitioning
   • Added chunk size configuration options

4. Case Handling Improvements

   • Added support for lower/upper case schema and table names
   • Improved temp table name handling for different database types
   • Added new format variables: stream_schema_lower/upper, stream_table_lower/upper

5. File System Updates

   • Enhanced file pattern matching across different file system implementations
   • Improved file deletion handling for SFTP
   • Better handling of file paths in copy operations

Minor Changes

• Added binary data handling for DuckDB
• Improved UTF-8 validation in string casting
• Added environment variable controls for DuckDB compute
• Various bug fixes and code cleanup