dyvenia/viadot v0.2.0

Viadot 0.2.0

on GitHub

This release brings many new tasks and flows, as well as improved stability, security, and monitoring.

Added

• Sources:

  • AzureDataLake (supports gen1 & gen2)
  • SQLite

• Tasks:

  • DownloadGitHubFile
  • AzureDataLakeDownload
  • AzureDataLakeUpload
  • AzureDataLakeToDF
  • ReadAzureKeyVaultSecret
  • CreateAzureKeyVaultSecret
  • DeleteAzureKeyVaultSecret
  • SQLiteInsert
  • SQLiteSQLtoDF
  • AzureSQLCreateTable
  • RunAzureSQLDBQuery
  • BCPTask
  • RunGreatExpectationsValidation
  • SupermetricsToDF

• Flows:

  • SupermetricsToAzureSQLv1
  • SupermetricsToAzureSQLv2
  • SupermetricsToAzureSQLv3
  • AzureSQLTransform
  • Pipeline
  • ADLSGen1ToGen2
  • ADLSGen1ToAzureSQL
  • ADLSGen1ToAzureSQLNew

• Examples:

  • Hello world flow
  • Supermetrics Google Ads extract

Changed

• Tasks now use secrets for credential management (azure tasks use Azure Key Vault secrets)
• SQL source now has a default query timeout of 1 hour

Fixed

• Fix SQLite tests
• Multiple stability improvements with retries and timeouts