vmware/versatile-data-kit v1.0.1 on GitHub

Major features include:

Secrets Service Helm Chart installation

Vault integration configuration for storing Data Job Secrets has been added to the Helm chart:

secrets:
    vault:
        enabled: false
        uri: "http://localhost:8200"
        
        externalSecretName: ""
        ## Alternatively provide the uri and Approle Settings here. externalSecretName takes precedence if both are set.
        approle:
            roleid: foo
            secretid: foo

        sizeLimitBytes: "1048576"

VDK Secrets CLI

Job secrets are used to store credentials/tokens/sensitive data securely. They can be updated using vdk-control-cli now:

Install vdk-control-cli if needed (it comes pre-installed in quickstart-vdk)

pip install vdk-control-cli

vdk secrets --help

For example:

     # Set single secret with key "my-key" and value "my-value". If no value is passed you'll get prompted so it's not printed on the screen.
     vdk secrets --set my-key "my-value"

     # Update multiple secrets at once.
     vdk secrets --set "key1" "value1" --set "key2" "value2" --set "secret1" --set "secret2"

Convert Directory-style To Notebook-style Data Job

With the introduction of Notebook-style data jobs, the user has the option to Convert Directory-style to Notebook-style Data Job.

VDK Menu	Pop-up window	Notebook

VDK Jupyter Extension published in PyPi

Users can now install the Jupyter extension with VDK in their own Python and jupyter environment with a single line :

pip install vdk-jupyterlab-extension

Then start Jupyter lab as usual:

jupyter lab

Users can now see the notebook:

Notebook

New plugin: vdk-smarter

VDK Smarter introduces proof of concept (pre-alpha) integration with OpenAI.

In the POC it does a review of all SQL queries managed by VDK.

For more details see the plugin home page

What's Changed

control-service: Add helm chart entries for Vault Configuation by @dakodakov in #2418
control-service: Update contributing.md with correct java requirements by @danail-georgiev in #2430
control-service: add configurable smtp host property by @mrMoZ1 in #2411
control-service: add helm template for alertmanager by @mrMoZ1 in #2326
control-service: add timestamps to helm chart by @DeltaMichael in #2344
control-service: better error logging for failed test by @murphp15 in #2374
control-service: fix helm chart by @dakodakov in #2449
control-service: fix publish-job-base-image script by @mivanov1988 in #2473
control-service: fix typo in helm chart read only root filesystem property by @mrMoZ1 in #2476
control-service: install necessary dependencies to job builder secure by @mivanov1988 in #2472
control-service: job-builder using kaniko fix by @tozka in #2429
control-service: job-builder-secure using kaniko fix by @tozka in #2447
control-service: logs endpoint doesn't hang by @murphp15 in #2370
control-service: prevent integer translation in helm chart by @dakodakov in #2470
control-service: push to multiple registries by @tozka in #2381
control-service: release job builder in 2 repos by @tozka in #2413
control-service: remove default vault token by @dakodakov in #2475
control-service: remove unused dependency influxdb by @tozka in #2388
control-service: run integration tests on multiple namespace. by @murphp15 in #2446
control-service: set Execution and JobQuery APIs to stable by @tozka in #2417
control-service: split build job base image CI/CD step by @mivanov1988 in #2348
control-service: switch to Approle Vault authentication by @dakodakov in #2435
control-service: use full url for heartbeat tests and heartbeat tests run in multiple namespaces by @murphp15 in #2295
frontend: Fix navigation in Data Jobs by @gorankokin in #2356
frontend: Fix router event handling in base class by @gorankokin in #2375
frontend: bump toolchain versions in frontend build docker image by @DeltaMichael in #2358
frontend: enable stable tagging by @DeltaMichael in #2378
frontend: fix data-pipelines build scripts by @DeltaMichael in #2389
frontend: push docker images to both repos by @tozka in #2390
frontend: quickstart-vdk operability tests using cypress by @DeltaMichael in #2359
frontend: remove e2e tests restrictions by @DeltaMichael in #2386
support: slack notification on pipeline failure by @DeltaMichael in #2338
vdk-control-cli: add vdk secrets command by @dakodakov in #2342
vdk-control-cli: add vdk secrets command by @dakodakov in #2357
vdk-control-cli: remove set-secret for properties by @dakodakov in #2409
vdk-core: Allow different python versions for vdk docker images by @doks5 in #2346
vdk-core: Set sender when checking if email exists by @doks5 in #2376
vdk-core: [Hot Fix] Stop throwing exceptions if config.ini not present by @doks5 in #2367
vdk-heartbeat: cover requirements.txt automatic installs by @tozka in #2393
vdk-impala: Truncate table before inserting data by @sbuldeev in #2369
vdk-impala: Update README.md for vdk-impala by @sbuldeev in #2355
vdk-impala: support also pydantic 1.0 by @tozka in #2368
vdk-impala: upgrade code to support pydantic 2.0 by @tozka in #2362
vdk-ipython: README.md fix by @duyguHsnHsn in #2345
vdk-jupyter: fix server error in jupyter ui and remove unneeded code by @duyguHsnHsn in #2361
vdk-jupyter: Add a message describing how to contact the Jupyter devs by @gageorgiev in #2414
vdk-jupyter: Create init cell when opening new notebook by @gageorgiev in #2352
vdk-jupyter: Sample job notebook step by @gageorgiev in #2364
vdk-jupyter: add Convert Job To Notebook UI button by @yonitoo in #2329
vdk-jupyter: convert job operation by @duyguHsnHsn in #2406
vdk-jupyter: publish image to pip registry by @murphp15 in #2407
vdk-jupyter: remove delete operation by @duyguHsnHsn in #2428
vdk-plugin-control-cli: add secrets command by @dakodakov in #2387
vdk-plugins: fix build of multiple plugins by @tozka in #2445
vdk-plugins: include Ingestion hooks documentation by @tozka in #2416
vdk-server: url in swagger ui is correct by @murphp15 in #2380
vdk-smarter: introduce simple open ai plugin by @tozka in #2351
versatile-data-kit: clarify structure of VDK repo by @tozka in #2432
versatile-data-kit: remove gitlint pre-commit hook by @tozka in #2360

New Contributors

@danail-georgiev made their first contribution in #2430

Full Changelog: v1.0...v1.0.1