github vmware/versatile-data-kit v1.3
Versatile Data Kit 1.3

latest release: v1.4
13 months ago

Major features include:

VDK SDK

Add vdk sql-query command (experimental)

CLI command to execute SQL query against VDK managed database.

It should replace vdk <db>-query commands.

export db_default_type=trino
vdk sql-query -q "select * from trino_table" 
id                                  memory_size_mb    num_vcpus
-------------- ----------------  -----------
50181506DB2F7               256            1
5018A2223FC32               128            1
501883404870A               256            1


vdk sql-query -o json -q "select * from trino_table" 
[
 {"id": "50181506DB2F7", "memory_size_mb": 256, "num_vcpus": 1}, 
 {"id": "5018A2223FC32", "memory_size_mb": 128, "num_vcpus": 1},
 {"id": "501883404870A", "memory_size_mb": 256, "num_vcpus": 1}
]

VDK Notebook Getting Started

Introduction to the development of VDK Jobs using Notebooks.

  • Learn how to create data jobs
  • Learn how to deploy data jobs

See more in https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-plugins/vdk-jupyter/getting-started.ipynb

VDK SQL Notebook Cell

image

VDK Errors APIs

(Relevant for plugin developers)
VDK is deprecating the user of errors.log_and_rethrow and errors.log_and_throw in favour of

errors.report(error_type, exception: BaseException)
errors.report_and_throw(exception: BaseVdkException)
errors.report_and_rethrow(error_type, exception: BaseException)

The aim is to reduce "double" logging and verbosity of logs.

Control Service

Add support for multiple jwt issuers

New properly security.oauth2.jwtIssuerUris is introduced and replaced jwtIssuerUrl

security:
   oauth2:
      
      ## [ Required if security.enabled = True ]
      ## Deprecated in favor of jwtIssuerUris.
      jwtIssuerUrl: ""
      ## [Required if security.enabled = True]
      ## Comma separated list of issuers to use.
      jwtIssuerUris: ""

Implement Webhook APIs authentication

Enable setting system or service account for Control Service webhook authentication

This introduces 2 new properties for each webhook

      authorizationServerEndpoint: ""
      authorizationRefreshToken: ""

If set they will be used when making HTTP Webhook request. If not it will fall back to the user provided authentication token

What's Changed

Full Changelog: v1.2...v1.3

Don't miss a new versatile-data-kit release

NewReleases is sending notifications on new releases.