Major features include:
Open-sourcing VDK Operations UI
VDK Operations UI would enable data practitioners to efficiently manage (operate and monitor) their data jobs.
It has been used internally in VMware for some time and the team open source it last month.
Check out more details at the Operations UI VEP
Look forward to the official launch soon.
Documentation Improvements
Significantly simplified and improve the main README and the CONTRIBUTING.md thanks to @gary-tai and @zverulacis
VDK Meta Jobs Preparation for Alpha release
implemented a limit on starting jobs at once
META_JOBS_MAX_CONCURRENT_RUNNING_JOBS=<number>
Learn more about the VDK Meta Jobs features in VDK Meta Jobs VEP
Started initiative to support multiple python versions
We are working on introducing an optional python_version property to the Control Service API, which allows users to specify the Python version they want to use for their job deployment. This means users no longer have to rely on the service administrator to make changes to the configuration and can deploy their jobs with the version they need.
See more information in the Multiple Python Versions VEP
Started initiative to create Secrets Interface
So far the way VDK recommended to store secrets was to use Properties API. Though it works well, it doesn't really meet the criteria for storing properly restricted data and likely also confidential data
The team is working on providing similar to Properties interface Secrets (backed by HashiCorp Vault).
See more information in the Vault Integration For Secrets Storage VEP
What's Changed
- control-service: Update helm template license headers by @DeltaMichael in #1670
- control-service: don't include needless service token by @tozka in #1679
- control-service: protect against v1 cron batch not being supported by @murphp15 in #1777
- docs: Documentation improvement: improve main README VDK images - gif by @zverulacis in #1676
- docs: Fix CICD build for control service by @murphp15 in #1790
- docs: Hotfix: Update the URL for gif by @zverulacis in #1702
- docs: Improve documentation: before and after image by @zverulacis in #1696
- docs: Issue 1574: Update Contributing document by @gary-tai in #1741
- docs: Update VEP-1507 with CI/CD description by @DeltaMichael in #1707
- docs: update contributor documentation by @DeltaMichael in #1779
- frontend: Create build and release jobs for vdk/shared package by @DeltaMichael in #1672
- frontend: Fix shebang in build_shared.sh by @DeltaMichael in #1743
- frontend: Rename vdk/shared to versatiledatakit/shared by @DeltaMichael in #1694
- frontend: bump data-pipelines dependency versions by @DeltaMichael in #1706
- frontend: compile fix by @ivakoleva in #1697
- frontend: data-pipelines build and test running CI/CD by @ivakoleva in #1671
- frontend: data-pipelines publish artifact CI/CD by @ivakoleva in #1675
- frontend: data-pipelines sync to freezepoint by @ivakoleva in #1746
- frontend: e2e tests (#6) by @ivakoleva in #1633
- frontend: e2e tests wiring and clientid by @ivakoleva in #1754
- frontend: gitlab ci variables for change locations by @ivakoleva in #1685
- frontend: package naming fixes by @ivakoleva in #1695
- specs: Add user journeys to VEP-1507 by @DeltaMichael in #1750
- specs: VDK Enhancement Proposal for Metajobs (complete) by @yonitoo in #1757
- specs: VEP-1493: Vault Integration For Secrets Storage by @dakodakov in #1756
- specs: VEP-1739 Support for Multiple Python Versions by @doks5 in #1748
- specs: edit vep-1507 for style and grammar by @DeltaMichael in #1758
- specs: vep-1416: Change status of vep by @doks5 in #1721
- vdk-audit-plugin: expand forbidden events list by @mivanov1988 in #1683
- vdk-heartbeat: fix test failures by @tozka in #1703
- vdk-impala: Add optional parameter for staging table prefix by @sbuldeev in #1666
- vdk-ipython: add finalize job by @duyguHsnHsn in #1744
- vdk-ipython: change the way job_input is introduced in a notebook by @duyguHsnHsn in #1678
- vdk-jupyter: jupyter UI refactor code and handle react-dom version mismatch by @duyguHsnHsn in #1727
- vdk-jupyter: Jupyter UI change the strategy on storing the data from user input by @duyguHsnHsn in #1624
- vdk-jupyter: add deploy job component to the UI by @duyguHsnHsn in #1704
- vdk-jupyter: add py-to-ts-interfaces to build by @duyguHsnHsn in #1765
- vdk-jupyter: load data before vdk operations by @duyguHsnHsn in #1762
- vdk-jupyter: remove py-to-ts-interfaces because of build problems by @duyguHsnHsn in #1691
- vdk-jupyter: rename python module by @duyguHsnHsn in #1784
- vdk-meta-jobs: Introduce configuration module by @gageorgiev in #1692
- vdk-meta-jobs: first version of the VEP Summary and Motivation sections by @yonitoo in #1698
- vdk-meta-jobs: implement a limit on starting jobs at once by @yonitoo in #1681
- vdk-meta-jobs: make _start_job() limit hit log more descriptive by @yonitoo in #1771
- vdk-plugins: bump docker version and resuse common dind by @tozka in #1701
- vdk-plugins: generate test report by @tozka in #1732
- vdk-plugins: upgrade docker used by @tozka in #1693
- vdk-quickstart: remove use of global varaibles in CI by @tozka in #1726
- vdk-trino: stabilize vdk-trino tests by @tozka in #1677
- versatile-data-kit: add hosts entry to gitlab runners by @ivakoleva in #1731
- versatile-data-kit: automate merging of dependabot PRs by @tozka in #1725
- versatile-data-kit: dependabot auto-merge fix by @ivakoleva in #1688
- versatile-data-kit: mechanism for targetted PR notifications. by @tozka in #1733
- versatile-data-kit: pre-commit hook for (S)CSS/JS/TS/HTML formatting by @ivakoleva in #1684
- versatile-data-kit: simplify release process by @tozka in #1668
New Contributors
Full Changelog: v0.11...v0.12