Major features include:
VDK DAG plugin release
VDK DAG (previously vdk-meta-jobs) is the official name of the plugin allowing users to express dependencies between data jobs and is released as Beta with more stability and usability and documentation improvements.
Check out for more in the plugin page.
Versatile Data Kit UI Shareable Web links
Now users can share links with filters applied:
- Data Jobs list (Manage and Explore screen) are shareable through URL, as every applied filter is persisted to URL and vice-versa
- Data Job Executions screen filters and sort parameters are shareable through URL, as every applied filter or sort is persisted to URL and vice-versa
VDK UI configuration improvements and easy to get started by using quickstart-vdk
Users can now access VDK UI using quickstart-vdk. VDK UI is made to be much more configurable:
- Toggleable authentication (default: enabled) using the 'skipAuth' flag.
- Configuration of authentication parameters.
- Ability to specify visual elements displayed, e.g., navigation button to the Explore page.
VDK Control CLI supports python version
People now can specify the python version they need their job to run when deployed in VDK Control Service runtime:
vdk deploy --python-version 3.7 ..
Or in job config.ini
[job]
python_version = 3.7
Users can also see which version of python is VDK Control Service supporting currently:
vdk info
would return something like
Getting control service information...
VDK Control service version: PipelinesControlService/0.0.1-SNAPSHOT/5f078fe ...
Supported python versions:
3.9
3.8
What's Changed
- control-service: Clean up old data job configurations by @doks5 in #2075
- control-service: Fix backwards-compatibility issues by @doks5 in #2022
- control-service: Only CLI executions are "Manual" by @gageorgiev in #1763
- control-service: Rework supported python version logic by @doks5 in #1992
- control-service: Swagger UI quickstart-vdk server config by @ivakoleva in #2062
- control-service: [Bug fix] Fix supported python versions helm configuration by @doks5 in #1964
- control-service: a clear error message on how to handle the failed pipeline by @murphp15 in #2127
- control-service: add ability to check if docker image exists in ecr by @mrMoZ1 in #1977
- control-service: allow more time to reach a complete state by @murphp15 in #2143
- control-service: append integration test name to job name by @mivanov1988 in #2093
- control-service: better error logging and pull private image in private test by @murphp15 in #2156
- control-service: better error message by @murphp15 in #2094
- control-service: better error message from throwable by @murphp15 in #2157
- control-service: clarify build steps by @dakodakov in #1959
- control-service: code expected to run in transaction now runs in transaction by @murphp15 in #2117
- control-service: custom app config values can we passed to helm. by @murphp15 in #2004
- control-service: delete unused method by @murphp15 in #2038
- control-service: disable authorization on test/cicd deployment by @tozka in #2129
- control-service: disable failing test by @murphp15 in #2086
- control-service: fail tests fast by @murphp15 in #2137
- control-service: fix api declaration by @murphp15 in #1974
- control-service: fix oom tests by @murphp15 in #2028
- control-service: handle null started by value by @murphp15 in #2151
- control-service: if a test is in a bad state it fails straight away by @murphp15 in #2098
- control-service: include details in error message by @murphp15 in #2122
- control-service: increase CICD deployment resources by @tozka in #2130
- control-service: killed job was shown as successful by @mivanov1988 in #2116
- control-service: latest version of gradle and spring /remove old comment by @murphp15 in #1976
- control-service: logs url can include team name by @murphp15 in #2013
- control-service: new python client. by @murphp15 in #1983
- control-service: print response body on error by @murphp15 in #2113
- control-service: remove a test that is testing behaviour that doesn't exist by @murphp15 in #2031
- control-service: remove unused parameter by @murphp15 in #2027
- control-service: remove unused parameters by @murphp15 in #2016
- control-service: see more details when there is an error by @murphp15 in #2050
- control-service: update ecr credentials integration test by @mrMoZ1 in #2079
- control-service: upgrade python client by @murphp15 in #2076
- control-service: use git for images by @murphp15 in #2097
- docs: add getting started section for quickstart-vdk and ui by @DeltaMichael in #2019
- frontend: Bugfix in e2e plugins function and bump major versions for UI libs by @gorankokin in #1994
- frontend: Fix for e2e tests by @gorankokin in #2030
- frontend: Implement executions list enhacements by @gorankokin in #2126
- frontend: Improve visibility of 'User error' messages by @hzhristova in #1960
- frontend: Job sharable executions filter and sort by @gorankokin in #2072
- frontend: Toggleable auth by @ivakoleva in #1958
- frontend: Upgrade lineage to beta version by @hzhristova in #1991
- frontend: shareable links with query params for Data Jobs grids by @hzhristova in #2049
- frontend: visibility of app components is configurable by @DeltaMichael in #1978
- quickstart-vdk: ignore explore page and widgets in frontend by @DeltaMichael in #2073
- vdk-airflow: fix failing tests by @murphp15 in #2078
- vdk-control-cli: Add support for python_version by @doks5 in #2002
- vdk-control-cli: Add support for python_version in config by @doks5 in #2023
- vdk-control-cli: add vdk info command to list of cli commands by @dakodakov in #2069
- vdk-control-cli: import the latest version of the client into cli by @murphp15 in #1969
- vdk-control-cli: upgrade python client by @murphp15 in #2077
- vdk-control-cli: use explicit parameter names by @murphp15 in #1975
- vdk-dag: DAGs propagate their execution type to their component jobs by @yonitoo in #2080
- vdk-dag: Drop deprecation warnings by @gageorgiev in #2012
- vdk-dag: Fix config bug by @gageorgiev in #2029
- vdk-dag: Rename vdk-meta-jobs to vdk-dag by @gageorgiev in #1831
- vdk-dag: fix plugin name of DAGs example README.md by @yonitoo in #1945
- vdk-dag: improve DAGs docs and example by @yonitoo in #1984
- vdk-dag: update VEP about the execution type propagation by @yonitoo in #2095
- vdk-examples: Change Meta Jobs to DAGs in examples by @gageorgiev in #2024
- vdk-gdp-execution-id: example added by @ivakoleva in #1962
- vdk-heartbeat: add test that pings the frontend url by @DeltaMichael in #2039
- vdk-heartbeat: fix python3.7 issue by @DeltaMichael in #2104
- vdk-impala: add udf hive2server user error classification by @mrMoZ1 in #2056
- vdk-jupyter: Drop faulty statement by @gageorgiev in #2160
- vdk-jupyter: Pick up REST API URL from the env by @gageorgiev in #2142
- vdk-jupyter: UI failing cell marks by @duyguHsnHsn in #2057
- vdk-jupyter: UI failing cell marks enhancement by @duyguHsnHsn in #2136
- vdk-jupyter: add separator to the VDK menu by @duyguHsnHsn in #2114
- vdk-jupyter: fix broken server in one use case by @duyguHsnHsn in #2170
- vdk-jupyter: pin lumino/widgets package to version 1 by @duyguHsnHsn in #2099
- vdk-jupyter: pin npm packages to previous version by @duyguHsnHsn in #2088
- vdk-meta-jobs: Undelete the package for potential future fixes by @gageorgiev in #2106
- vdk-server: logs url in quickstart vdk by @murphp15 in #2018
- vdk-server: trigger package release by @murphp15 in #2058
- versatile-data-kit: Update CONTRIBUTING.md by @tozka in #1988
New Contributors
- @hzhristova made their first contribution in #1960
Full Changelog: v0.13...v0.14