Added
- Added
adls_file_name
inSupermetricsToADLS
andSharepointToADLS
flows - Added
BigQueryToADLS
flow class which anables extract data from BigQuery. - Added
Salesforce
source - Added
SalesforceUpsert
task - Added
SalesforceBulkUpsert
task - Added C4C secret handling to
CloudForCustomersReportToADLS
flow (c4c_credentials_secret
parameter)
Fixed
- Fixed
get_flow_last_run_date()
incorrectly parsing the date - Fixed C4C secret handling (tasks now correctly read the secret as the credentials, rather than assuming the secret is a container for credentials for all environments and trying to access specific key inside it). In other words, tasks now assume the secret holds credentials, rather than a dict of the form
{env: credentials, env2: credentials2}
- Fixed
utils.gen_bulk_insert_query_from_df()
failing with > 1000 rows due to INSERT clause limit by chunking the data into multiple INSERTs - Fixed
get_flow_last_run_date()
incorrectly parsing the date - Fixed
MultipleFlows
when one flow is passed and when last flow fails.