Summary
Major features include:
Standalone Data Job run
Until now the only way to run a data job was with CLI command "vdk run". Now users can run a job entirely programmatically using it.
For example:
with StandaloneDataJobFactory.create(
data_job_directory=Path(__file__), extra_plugins=[hook_tracker]
) as job_input:
print(job_input.get_name())
Check out more in the new API documentation here
New Plugin: vdk-control-api-auth
A new library plugin, not a runnable plugin, that is intended to be used as a dependency for other plugins, which need to authenticate users against the Control Service.
Check more in the plugin documentation
What's Changed
- Scenario 3 - Created the Energy Scenario by @alod83 in #781
- [vdk-plugins] vdk-control-api-auth: Add api-token flow by @doks5 in #822
- [vdk-plugins] vdk-control-api-auth: Add authorization code flow by @doks5 in #834
- [vdk-plugins] vdk-control-api-auth: Enable plugin release by @doks5 in #837
- [vdk-plugins] vdk-control-api-auth: Fix query key type by @doks5 in #838
- airflow-provider-vdk: Fix CICD release step by @gageorgiev in #824
- airflow-provider-vdk: VDKOperator execute method by @gageorgiev in #823
- airflow-provider-vdk: VDKOperator initial structure by @gageorgiev in #820
- airflow-provider-vdk: VDKSensor poke method by @gageorgiev in #818
- control-service: Allow for jobs with no schedule to be deployed by @gageorgiev in #835
- control-service: Kerberos authentication IT by @mrMoZ1 in #798
- control-service: cicd unit tests should run on pull requests by @mrMoZ1 in #830
- control-service: kerberos authentication IT test by @mrMoZ1 in #831
- vdk-control-api-auth: Add core auth logic by @doks5 in #815
- vdk-control-cli: Adopt new vdk-control-api-auth library by @doks5 in #840
- vdk-control-cli: Fix command printed on successful deploy by @gageorgiev in #839
- vdk-control-cli: Make schedule_cron config param optional by @gageorgiev in #827
- vdk-core: New feature: StandaloneDataJob by @mrdavidlaing in #793
- vdk-core: encapsulate router-specific properties logic by @ivakoleva in #817
- vdk-core: new version check built-in plugin false positive fix by @ivakoleva in #816
- vdk-core: properties write pre-processing support by @ivakoleva in #819
- vdk-heartbeat: null datetime conversion fix by @ivakoleva in #813
- versatile-data-kit: allow commit with any newer version of python by @tozka in #826
- versatile-data-kit: link examples wiki in the git examples by @tozka in #812
- versatile-data-kit: update readme with clear slack instructions by @tozka in #806
New Contributors
- @mrdavidlaing made their first contribution in #793
Full Changelog: v0.3...v0.4