Major features include:
New plugin: vdk-gdp-execution-id
An installed Generative Data Pack plugin automatically expands the data sent for ingestion.
This GDP plugin detects the execution ID of a Data Job running, and decorates your data product with it. So that,
it is now possible to correlate a data record with a particular ingestion Data Job execution ID.
For more information see the plugin documentation
vdk-dag: pass arguments to jobs in a DAG
Now each job in a DAG can be passed arguments :
{
"job_name": "name-of-job",
"team_name": "team-of-job",
"fail_meta_job_on_error": false,
"arguments": <ARGUMENTS IN DICTIONARY FORMAT HERE>,
"depends_on": ["name-of-job1", "name-of-job2"]
}vdk-notebook: VDK job input in vdk cells
Users will be able to develop jobs entirely in a Notebook file with all features of VDK available out of the box
After installation of vdk-notebook users can now will have access to job_input interface to execute templates, ingest data and all else.
vdk-notebook: vdk and non-vdk cells
To enable separation of product and development code vdk-notebook integration provides a way for users to set which cells are deployable and part of their production code and which are not.
quickstart-vdk now includes the Operations UI
When installing quickstart-vdk VDK Server is available for local testing and now includes UI:
pip install quickstart-vdk
vdk server --install
For more information see here
Versatile Data Kit Frontend npm libraries release
The Versatile Data Kit Frontend provides 2 npm (angular) libraries which can be used to build integrate VDK UI with your own screens:
- @versatiledatakit/data-pipelines
Versatile Data Kit Data Pipelines library provides UI screens that helps to manage data jobs via Versatile Data Kit Control Service - @versatiledatakit/shared
Versatile Data Kit Shared library enables reusability of shared features like: NgRx Redux, Error Handlers, Utils, Generic Components, etc.
What's Changed
- control service: Add supported python version configuration by @doks5 in #1761
- control-service: fix python api release by @murphp15 in #1946
- control service: Dynamically set job base image in builder by @doks5 in #1864
- control-service: Add python_version to Control Service API by @doks5 in #1806
- control-service: Add python_version to Execution API by @mivanov1988 in #1878
- control-service: Add python_version to GraphQL API by @mivanov1988 in #1909
- control-service: Add support for Python 3.11 by @mivanov1988 in #1861
- control-service: Dynamically set vdk image in JobImageDeployer by @doks5 in #1883
- control-service: Expose supported python versions in helm by @doks5 in #1935
- control-service: Remove support for very old k8s apiVersion by @murphp15 in #1860
- control-service: add the frontend to helm by @murphp15 in #1885
- control-service: enable usage of aws temporary credentials by @mrMoZ1 in #1787
- control-service: expose supported python versions by @dakodakov in #1841
- control-service: fix failing image publisher by @murphp15 in #1810
- control-service: force job builder deploy by @mrMoZ1 in #1823
- control-service: new helm release by @murphp15 in #1910
- control-service: revert job builder python version by @mrMoZ1 in #1840
- control-service: update helm charts for service account credentials by @mrMoZ1 in #1800
- control-service: update job builders for aws temporary credentials by @mrMoZ1 in #1799
- documentation: VDK components explained by @ivakoleva in #1865
- frontend: Align code formatting in frontend projects by @gorankokin in #1863
- frontend: Configurable OAuth by @ivakoleva in #1913
- frontend: Update docs with build/test configuration by @DeltaMichael in #1928
- frontend: add build.sh by @tozka in #1807
- frontend: fix npm lint warnings by @DeltaMichael in #1808
- frontend: increase the amount of resources for build in cicd by @murphp15 in #1931
- frontend: prepare for official release shared and dp libs by @gorankokin in #1795
- frontend: publish docker image for ui by @DeltaMichael in #1872
- frontend: remove unused config in helm chart for frontend dns by @murphp15 in #1932
- frontend: Stabilization for e2e tests by @gorankokin in #1876
- frontend: Auth configurations organized by @ivakoleva in #1957
- frontend: change history link in data job by @gorankokin in #1884
- specs: VEP-1739 Update status and reorganise document by @doks5 in #1857
- specs: VEP-1739 updated API section by @mivanov1988 in #1882
- specs: update Multiple Python versions VEP summary by @tozka in #1792
- vdk-vep: update vep status by @dakodakov in #1951
- vdk-cicd: apply limit ranges for storage by @tozka in #1815
- vdk-cicd: set ephemeral storage request/limits by @tozka in #1813
- vdk-control-cli: fix circular import dependecy by @tozka in #1820
- vdk-control-cli: refactor output printing with printer class by @tozka in #1819
- vdk-control-cli: use assert_click_status by @tozka in #1817
- vdk-control-cli: use common output printer by @tozka in #1852
- vdk-control-cli: vdk list -mmm to return executions by @tozka in #1818
- vdk-control-service: publish python client library by @dakodakov in #1934
- vdk-dags: improve DAGs user-facing documentation by @yonitoo in #1892
- vdk-gdp-execution-id: a Generative Data Pack expanding with execution ID by @ivakoleva in #1877
- vdk-gdp-execution-id: import fix by @ivakoleva in #1961
- vdk-github-workflows: ubuntu latest update by @ivakoleva in #1943
- vdk-jupyter: UI test enhancements by @duyguHsnHsn in #1783
- vdk-jupyter: add UI vdk cell marks by @duyguHsnHsn in #1891
- vdk-jupyter: job run messages by @duyguHsnHsn in #1908
- vdk-jupyter: remove react-test-renderer package from package.json by @duyguHsnHsn in #1881
- vdk-lineage: support for latest version sqllineage library by @tozka in #1816
- vdk-meta-jobs: Meta Jobs DAG validation by @yonitoo in #1785
- vdk-meta-jobs: add DAG with args example by @yonitoo in #1859
- vdk-meta-jobs: add some configurable variable references in the VEP by @yonitoo in #1794
- vdk-meta-jobs: exec job with arguments by @yonitoo in #1839
- vdk-meta-jobs: fix DAG image in example by @yonitoo in #1920
- vdk-meta-jobs: improve DAGs code documentation by @yonitoo in #1873
- vdk-metajobs: Deprecate plugin by @gageorgiev in #1930
- vdk-notebook: add hook for saving error information into json file by @duyguHsnHsn in #1842
- vdk-plugins: add connection hook activity diagram by @tozka in #1786
- vdk-plugins: test only oldest and newest supported python version by @tozka in #1811
- vdk-server: quickstart vdk now includes the UI by @murphp15 in #1912
New Contributors
- @gorankokin made their first contribution in #1795
Full Changelog: v0.12...v0.13