dyvenia/viadot v0.4.0

Viadot 0.4.0

on GitHub

Added

• Added custom_mail_state_handler function that sends mail notification using custom smtp server.
• Added new function df_clean_column that cleans data frame columns from special characters
• Added df_clean_column util task that removes special characters from a pandas DataFrame
• Added MultipleFlows flow class which enables running multiple flows in a given order.
• Added GetFlowNewDateRange task to change date range based on Prefect flows
• Added check_col_order parameter in ADLSToAzureSQL
• Added new source ASElite
• Added KeyVault support in CloudForCustomers tasks
• Added SQLServer source
• Added DuckDBToDF task
• Added DuckDBTransform flow
• Added SQLServerCreateTable task
• Added credentials param to BCPTask
• Added get_sql_dtypes_from_df and update_dict util tasks
• Added DuckDBToSQLServer flow
• Added if_exists="append" option to DuckDB.create_table_from_parquet()
• Added get_flow_last_run_date util function
• Added df_to_dataset task util for writing DataFrames to data lakes using pyarrow
• Added retries to Cloud for Customers tasks
• Added chunksize parameter to C4CToDF task to allow pulling data in chunks
• Added chunksize parameter to BCPTask task to allow more control over the load process
• Added support for SQL Server's custom datetimeoffset type
• Added AzureSQLToDF task
• Added AzureSQLUpsert task

Changed

• Changed the base class of AzureSQL to SQLServer
• df_to_parquet() task now creates directories if needed
• Added several more separators to check for automatically in SAPRFC.to_df()
• Upgraded duckdb version to 0.3.2

Fixed

• Fixed bug with CheckColumnOrder task
• Fixed OpenSSL config for old SQL Servers still using TLS < 1.2
• BCPTask now correctly handles custom SQL Server port
• Fixed SAPRFC.to_df() ignoring user-specified separator
• Fixed temporary CSV generated by the DuckDBToSQLServer flow not being cleaned up
• Fixed some mappings in get_sql_dtypes_from_df() and optimized performance
• Fixed BCPTask - the case when the file path contained a space
• Fixed credential evaluation logic (credentials is now evaluated before config_key)
• Fixed "$top" and "$skip" values being ignored by C4CToDF task if provided in the params parameter
• Fixed SQL.to_df() incorrectly handling queries that begin with whitespace

Removed

• Removed autopick_sep parameter from SAPRFC functions. The separator is now always picked automatically if not provided.
• Removed dtypes_to_json task to task_utils.py