github kestra-io/kestra v0.10.0
on GitHub

latest releases: v0.19.10, v0.15.24, v0.18.12...
16 months ago

Kestra Release Note - 0.10.0

This new release brings important new core features such as Blueprints, new Script plugins, integration of basic authentication, and a secret function making the Open Source Edition even more robust.

Features

Blueprints

Blueprints are a curated, organized, and searchable catalog of ready-to-use examples designed to help you kick-start your workflow.

It’s available as a new section on the left bar menu and directly within the editor view**. All blueprints are validated and documented so that they just work.

You can easily customize and integrate them into your new or existing flows with a single click on the “Use” button.

Check out the documentation.

Untitled

Untitled2

Enterprise Edition users could write their own organization blueprints so they can share and grow a library of internal blueprints within teams.
Untitled3

Improved Support for Scripting

In this release, we revamped scripting tasks to bring more flexibility and control.
Each script task is now, by default, running in its own dedicated container.You can attach any Docker image you want if you need a specific one.
New tasks are available for Shell, Python, R, and Node.js:

  • Script: to write and run ad-hoc scripts. It comes with a beforeCommands property to execute any instruction needed before running the main script (installing dependencies for example) 
    id: script
    namespace: release
    
    tasks:
      - id: run_python
        type: io.kestra.plugin.scripts.python.Script
        beforeCommands:
          - pip install requests
        warningOnStdErr: false
        script: | 
          import requests
          import json
    
          response = requests.get("https://api.github.com")
          data = response.json()
          print(data)
  • Command: to run arbitrary commands in a single task configuration. This task can be very powerful and associated with the WorkingDirectory task. For example, to clone a Git repository and then execute the corresponding scripts. 
    id: command
    namespace: release
    
    tasks:
      - id: working
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
    
          - id: clone_repo
            type: io.kestra.plugin.git.Clone
            url: https://github.com/your_repository
    
          - id: run_python
            type: io.kestra.plugin.scripts.python.Command
            beforeCommands:
              - pip install -r requirements.txt
        dockerOptions:
        	    image: ghcr.io/kestra-io/pydata:latest
            commands:
              - python main.py

  • LocalFile: to create files in the local filesystem or to send files from the local filesystem to the internal storage. This task allows to uncouple inline scripts to their execution. 

        id: "local-file"
    namespace: release
    tasks:
      - id: workingDir
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
        - id: inputFiles
          type: io.kestra.core.tasks.storages.LocalFiles
          inputs:
            hello.txt: "Hello World"
            address.json: "{{ outputs.myTaskId.uri }}"
        - id: bash
          type: io.kestra.plugin.scripts.shell.Command
          commands:
            - cat hello.txt
    	- sed -n 's/.*"country":"\([^"]*\)".*/\1/p' address.json

It also allows to expose outputs to internal storage. In the following example, we create two files with Bash commands and expose those into Kestra internal storage with the outputs property LocalFiles 

    id: "local-files"
    namespace: release
    
    tasks:
      - id: workingDir
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
        - id: bash
          type: io.kestra.plugin.scripts.shell.Command
          commands:
            - mkdir -p sub/dir
            - echo "Hello from Bash" >> sub/dir/bash1.txt
            - echo "Hello from Bash" >> sub/dir/bash2.txt
        - id: outputFiles
          type: io.kestra.core.tasks.storages.LocalFiles
          outputs:
            - sub/**

Those tasks run by default on DOCKER but you can use the runner: PROCESS property to run it as a process on the Kestra host.

Note: the old scripting tasks w ill be deprecated, removed from the core and being retro compatible within the new plugins.

DAG task

Creating directed acyclic graphs, a common pattern in data orchestration, was already possible in Kestra by through Flow dependencies. With a brand new DAG task, it’s now even easier to do; directly between tasks at the Flow level. 

id: magicDAG
namespace: dev
tasks:
  - id: dag
    type: io.kestra.core.tasks.flows.Dag
    tasks:
      - task:
          id: customers
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_customers.csv
      - task:
          id: orders
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_orders.csv
      - task:
          id: payments
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_payments.csv
      - task:
          id: transform
          type: io.kestra.core.tasks.log.Log
          message: |
            Transforming data from: 
              {{outputs.customers.uri}}
              {{outputs.orders.uri}}
              {{outputs.payments.uri}}
        dependsOn:
          - customers
          - orders
          - payments

image

Label on executions

So far, it was only possible to add labels on a flow level by adjusting the workflow code. This release adds the ability to set custom labels for specific Executions.

Also, the labels added on a flow level will be automatically propagated to Execution labels.

For example you can add labels for “experiment” executions so you can retrieve them in the UI easily.
Untitled4

Screenshot 2023-07-05 at 15 43 45

Basic Authentication

The community was asking about authentication in the open source to secure their instance in production. We heard you ! You can now add basic authentication to your Kestra instance with username/password values in the Kestra configuration. 

server:
  basic-auth:
    enabled: true
    username: admin
    password: *****

Secret Function

We introduced a secret() function allowing to read base64 secret value from environment variables. 

id: secret
namespace: release
tasks:
  - id: get_secret
    type: io.kestra.core.tasks.debugs.Return
    format: '{{ secret("my_secret")}}'

Worker Group

This release introduce worker groups, a ****set of workers that can be targeted specifically for a task execution or a polling trigger evaluation. For this, the task or the polling trigger must define the workerGroup.key property with the key of the worker group to target. A default worker group can also be configured at the namespace level.

Here are common use cases in which Worker Groups can be beneficial:

  • Execute tasks and polling triggers on specific compute instances (e.g., a VM with a GPU and preconfigured CUDA drivers).
  • Execute tasks and polling triggers on a worker with a specific Operating System (e.g., a Windows server).
  • Restrict backend access to a set of workers (firewall rules, private networks, ...).
  • Execute tasks and polling triggers close to a remote backend (region selection).

Here is how you can ensure that a task is executed only by specific worker instances: 

id: gpuTask
namespace: dev
tasks:
  - id: hello
    type: io.kestra.core.tasks.log.Log
    message: |
      This task will be executed on a specific remote worker that has access to a GPU
    workerGroup:
    key: gpuWorkerGroupKey

Polling Triggers

We have made significant improvements to the Polling trigger in the latest version of Kestra to enhance performance, strengthen security measures, simplify maintenance, and clarify the system architecture.
In previous versions, the trigger evaluation process was directly handled by the Scheduler. This meant that both the Worker and the Scheduler potentially accessed external systems, leading to manage the security and scalability of two services.

From now on, The Scheduler delegates the evaluation of polling triggers to the Worker. This architectural change has several advantages. Firstly, it simplifies the overall system architecture by ensuring that only the Worker needs to interact with external systems. This separation of responsibilities streamlines the flow of data and improves the overall system performance.
By reducing the direct access of the Scheduler to external systems, we also minimize potential security vulnerabilities and fortify the protection of sensitive data.

Plugins

All Changes

Bug Fixes

Documentation

Code Refactoring

Tests

Chores

Commits

  • 57569bf: chore(github) : add a blueprint issue template (#1539) (Anna Geller)
  • 57a67d4: Merge branch 'release' into develop (YannC)
  • 9367ef3: feat(ui+webserver): Blueprint tags filtered by query on blueprints (#1667) (brian-mulier-p) #1667
  • 97fbc93: feat/editor-validation-feedback-rework (#1662) (brian-mulier-p) #1662
  • ed80eb5: fix/non-b64-secrets-prevent-crash (#1711) (brian-mulier-p) #1711
Check out latest releases or
releases around kestra-io/kestra v0.10.0

Don't miss a new kestra release

NewReleases is sending notifications on new releases.

Get notifications