What's Changed
- deletes all pipeline state (schemas, state, intermediate files) if destination dataset is dropped
- synchronizes state with the destination in the
runmethod - you can opt out from state sync with
restore_from_destination=falsein ie.config.toml - loads all schemas/source into single dataset by default. this will simplify experience for less advanced users. you can switch back to the old behavior (each source/schema has separate dataset) with
use_single_dataset=falseconfig option - enables CTRL-C when running user code
- commits all files extracted from several sources after all user code ran