Major features include:
VDK SDK
Add vdk sql-query command (experimental)
CLI command to execute SQL query against VDK managed database.
It should replace vdk <db>-query commands.
export db_default_type=trino
vdk sql-query -q "select * from trino_table"
id memory_size_mb num_vcpus
-------------- ---------------- -----------
50181506DB2F7 256 1
5018A2223FC32 128 1
501883404870A 256 1
vdk sql-query -o json -q "select * from trino_table"
[
{"id": "50181506DB2F7", "memory_size_mb": 256, "num_vcpus": 1},
{"id": "5018A2223FC32", "memory_size_mb": 128, "num_vcpus": 1},
{"id": "501883404870A", "memory_size_mb": 256, "num_vcpus": 1}
]
VDK Notebook Getting Started
Introduction to the development of VDK Jobs using Notebooks.
- Learn how to create data jobs
- Learn how to deploy data jobs
VDK SQL Notebook Cell
VDK Errors APIs
(Relevant for plugin developers)
VDK is deprecating the user of errors.log_and_rethrow and errors.log_and_throw in favour of
errors.report(error_type, exception: BaseException)
errors.report_and_throw(exception: BaseVdkException)
errors.report_and_rethrow(error_type, exception: BaseException)
The aim is to reduce "double" logging and verbosity of logs.
Control Service
Add support for multiple jwt issuers
New properly security.oauth2.jwtIssuerUris is introduced and replaced jwtIssuerUrl
security:
oauth2:
## [ Required if security.enabled = True ]
## Deprecated in favor of jwtIssuerUris.
jwtIssuerUrl: ""
## [Required if security.enabled = True]
## Comma separated list of issuers to use.
jwtIssuerUris: ""
Implement Webhook APIs authentication
Enable setting system or service account for Control Service webhook authentication
This introduces 2 new properties for each webhook
authorizationServerEndpoint: ""
authorizationRefreshToken: ""
If set they will be used when making HTTP Webhook request. If not it will fall back to the user provided authentication token
What's Changed
- control-service: add amazon rds ca certificates to data job image by @mrMoZ1 in #2660
- control-service: add data job deployment dynamic property source selection mechanism by @mrMoZ1 in #2641
- control-service: add missing packages to secure job builder by @mivanov1988 in #2662
- control-service: add support for multiple jwt issuers by @mrMoZ1 in #2628
- control-service: data jobs synchronizer initial implementation by @mivanov1988 in #2633
- control-service: implement Webhook APIs authentication by @mivanov1988 in #2655
- control-service: introduce data job deployment entity by @mivanov1988 in #2659
- control-service: webhook authentication comments by @mivanov1988 in #2663
- specs: VDK Run Logs: add detailed design for Progress tracker functionality by @antoniivanov in #2647
- specs: VEP-2421 Universal Database Plugin by @Maximiliaan72 in #2616
- vdk-control-api-auth: [bug-fix] authorization header base64 encoding by @doks5 in #2658
- vdk-control-cli: download keytab along with downloading job by @duyguHsnHsn in #2617
- vdk-core: add log stacktrace flag by @DeltaMichael in #2648
- vdk-core: add more unit tests to for job_input_error_classfier by @DeltaMichael in #2621
- vdk-core: add vdk sql-query command by @antoniivanov in #2649
- vdk-core: change jobs behavior when logging unavailable by @mrMoZ1 in #2656
- vdk-core: provide more info about the machine vdk is running on by @antoniivanov in #2629
- vdk-core: remove box around exception when printed by @DeltaMichael in #2700
- vdk-core: separate error logging, error reporting and error classification by @DeltaMichael in #2666
- vdk-csv: uncomment test by @duyguHsnHsn in #2704
- vdk-duckdb: update readme by @antoniivanov in #2657
- vdk-heartbeat: Add python version configuration by @doks5 in #2675
- vdk-ipython: add support for %%vdksql magic command by @antoniivanov in #2619
- vdk-jupyter: Change root FS position for path parameter by @gageorgiev in #2550
- vdk-jupyter: Log file is not refreshed on new vdk run by @duyguHsnHsn in #2687
- vdk-jupyter: Update ConvertJobToNotebook doc by @antoniivanov in #2702
- vdk-jupyter: VDK menu fix by @duyguHsnHsn in #2635
- vdk-jupyter: add first version of getting started page by @duyguHsnHsn in #2516
- vdk-jupyter: add opionated jupyter extension as extra by @antoniivanov in #2625
- vdk-jupyter: change Create and Download default path by @duyguHsnHsn in #2669
- vdk-jupyter: fix arguments input size by @duyguHsnHsn in #2685
- vdk-jupyter: fix conversion by @duyguHsnHsn in #2699
- vdk-jupyter: improve init message by @antoniivanov in #2688
- vdk-jupyter: pin traitlets to 5.9 by @antoniivanov in #2674
- vdk-jupyter: rename log file by @duyguHsnHsn in #2703
- vdk-jupyter: show status on starting new operation by @antoniivanov in #2627
- vdk-jupyter: use npm in build.sh by @antoniivanov in #2665
- vdk-jupyter: show status dialog on deploy and create by @antoniivanov in #2631
- vdk-notebook: add support for execute %%vdksql cells by @antoniivanov in #2622
- versatile-data-kit: CONTRIBUTING.md feedback / improvements #2146 by @zverulacis in #2661
- versatile-data-kit: README user testing improvements by @zverulacis in #2654
Full Changelog: v1.2...v1.3