dagster-io/dagster 1.3.10

1.3.10 (core) / 0.19.10 (libraries)

on GitHub

New

• [dagster-dbt] By default, freshness policies and auto materialize policies on dbt assets can now be specified using the dagster field under +meta configuration. The following are equivalent:

Before:

version: 2

models:
  - name: users
    config:
      dagster_freshness_policy:
        maximum_lag_minutes: 60
        cron_schedule: '0 9 * * *'
      dagster_auto_materialize_policy:
        type: 'lazy'

After:

version: 2

models:
  - name: users
    config:
      meta:
        dagster:
          freshness_policy:
            maximum_lag_minutes: 60
            cron_schedule: '0 9 * * *'
          auto_materialize_policy:
            type: 'lazy'

• Added support for Pythonic Config classes to the @configured API, which makes reusing op and asset definitions easier:

  class GreetingConfig(Config):
      message: str
  
  @op
  def greeting_op(config: GreetingConfig):
      print(config.message)
  
  class HelloConfig(Config):
      name: str
  
  @configured(greeting_op)
  def hello_op(config: HelloConfig):
      return GreetingConfig(message=f"Hello, {config.name}!")

• Added AssetExecutionContext to replace OpExecutionContext as the context object passed in to @asset functions.

• TimeWindowPartitionMapping now contains an allow_nonexistent_upstream_partitions argument that, when set to True, allows a downstream partition subset to have nonexistent upstream parents.

• Unpinned the alembic dependency in the dagster package.

• [ui] A new “Assets” tab is available from the Overview page.

• [ui] The Backfills table now includes links to the assets that were targeted by the backfill.

Bugfixes

• Dagster is now compatible with a breaking change introduced in croniter==1.4.0. Users of earlier versions of Dagster can pin croniter<1.4.
• Fixed an issue introduced in 1.3.8 which prevented resources from being bound to sensors when the specified job required late-bound resources.
• Fixed an issue which prevented specifying resource requirements on a @run_failure_sensor.
• Fixed an issue where the asset reconciliation sensor failed with a “invalid upstream partitions” error when evaluating time partitions definitions with different start times.
• [dagster-k8s] Fixed an issue where annotations are not included in the Dagster Helm chart for the pod that is created when configuring the Helm chart to run database migrations.
• [ui] Fixed an issue with filtering runs by created date on the Runs page.
• [ui] The “upstream partitions missing” warning no longer appears in the asset backfill dialog if the upstream partitioned asset is a source asset.
• [dagster-dbt] Fixed an issue where asset dependencies for dbt models with ephemeral models in between them would sometimes be improperly rendered.

Community Contributions

• Added support for setting resources in asset and multi_asset sensors. Thanks @plaflamme!
• Fixed an issue where py.typed was missing in the dagster-graphql package. Thanks @Tanguy-LeFloch!

Experimental

• Evaluation history for AutoMaterializePolicys will now be cleared after 1 week.
• [dagster-dbt] Several improvements to @dbt_assets:
  • profile and target can now be customized on the DbtCli resource.
  • If a partial_parse.msgpack is detected in the target directory of your dbt project, it is now copied into the target directories created by DbtCli to take advantage of partial parsing.
  • The metadata of assets generated by @dbt_assets can now be customized by overriding DbtManifest.node_info_to_metadata.
  • Execution duration of dbt models is now added as default metadata to AssetMaterializations.

Documentation

• Added a new tutorial section about using resources.

Dagster Cloud

• Fixed an issue where overriding the container name of a code server pod using serverK8sConfig.containerConfig.name did not actually change the container name.