dyvenia/viadot v0.4.6

Viadot 0.4.6

on GitHub

Added

• Added rfc_character_limit parameter in SAPRFCToDF task, SAPRFC source, SAPRFCToADLS and SAPToDuckDB flows
• Added on_bcp_error and bcp_error_log_path parameters in BCPTask
• Added ability to process queries which result exceed SAP's character per low limit in SAPRFC source
• Added new flow PrefectLogs for extracting all logs from Prefect with details
• Added PrefectLogs flow

Changed

• Changed CheckColumnOrder task and ADLSToAzureSQL flow to handle appending to non existing table
• Changed tasks order in EpicorOrdersToDuckDB, SAPToDuckDB and SQLServerToDuckDB - casting
  DF to string before adding metadata
• Changed add_ingestion_metadata_task() to not to add metadata column when input DataFrame is empty
• Changed check_if_empty_file() logic according to changes in add_ingestion_metadata_task()
• Changed accepted values of if_empty parameter in DuckDBCreateTableFromParquet
• Updated .gitignore to ignore files with *.bak extension and to ignore credentials.json in any directory
• Changed logger messages in AzureDataLakeRemove task

Fixed

• Fixed handling empty response in SAPRFC source
• Fixed issue in BCPTask when log file couln't be opened.
• Fixed log being printed too early in Salesforce source, which would sometimes cause a KeyError
• raise_on_error now behaves correctly in upsert() when receiving incorrect return codes from Salesforce

Removed

• Removed option to run multiple queries in SAPRFCToADLS