dagster-io/dagster 1.2.4

1.2.4 (core) / 0.18.4 (libraries)

on GitHub

New

• Further performance improvements to the asset reconciliation sensor.
• Performance improvements to asset backfills with large numbers of partitions.
• New AssetsDefinition.to_source_assets to method convert a set of assets to SourceAsset objects.
• (experimental) Added partition mapping that defines dependency relationships between different MultiPartitionsDefinitions.
• [dagster-mlflow] Removed the mlflow pin from the dagster-mlflow package.
• [ui] Syntax highlighting now supported in rendered markdown code blocks (from metadata).

Bugfixes

• When using build_asset_reconciliation_sensor, in some cases duplicate runs could be produced for the same partition of an asset. This has been fixed.

• When using Pythonic configuration for resources, aliased field names would cause an error. This has been fixed.

• Fixed an issue where context.asset_partitions_time_window_for_output threw an error when an asset was directly invoked with build_op_context.

• [dagster-dbt] In some cases, use of ephemeral dbt models could cause the dagster representation of the dbt dependency graph to become incorrect. This has been fixed.

• [celery-k8s] Fixed a bug that caused JSON deserialization errors when an Op or Asset emitted JSON that doesn't represent a DagsterEvent.

• Fixed an issue where launching a large backfill while running dagster dev would sometimes fail with a connection error after running for a few minutes.

• Fixed an issue where dagster dev would sometimes hang when running Dagster code that attempted to read in input via stdin.

• Fixed an issue where runs that take a long time to import code would sometimes continue running even after they were stopped by run monitoring for taking too long to start.

• Fixed an issue where AssetSelection.groups() would simultaneously select both source and regular assets and consequently raise an error.

• Fixed an issue where BindResourcesToJobs would raise errors encapsulating jobs which had config specified at definition-time.

• Fixed Pythonic config objects erroring when omitting optional values rather than specifying None.

• Fixed Pythonic config and resources not supporting Enum values.

• DagsterInstance.local_temp and DagsterInstance.ephemeral now use object instance scoped local artifact storage temporary directories instead of a shared process scoped one, removing a class of thread safety errors that could manifest on initialization.

• Improved direct invocation behavior for ops and assets which specify resource dependencies as parameters, for instance:

  class MyResource(ConfigurableResource):
      pass
  
  @op
  def my_op(x: int, y: int, my_resource: MyResource) -> int:
      return x + y
  
  my_op(4, 5, my_resource=MyResource())

• [dagster-azure] Fixed an issue with an AttributeError being thrown when using the async DefaultAzureCredential (thanks @mpicard)

• [ui] Fixed an issue introduced in 1.2.3 in which no log levels were selected by default when viewing Run logs, which made it appear as if there were no logs at all.

Deprecations

• The environment_vars argument to ScheduleDefinition is deprecated (the argument is currently non-functional; environment variables no longer need to be whitelisted for schedules)

Community Contributions

• Typos fixed in [CHANGES.md](http://CHANGES.md) (thanks @fridiculous)
• Links to telemetry docs fixed (thanks @Abbe98)
• --path-prefix can now be supplied via Helm chart (thanks @mpicard)

Documentation

• New machine learning pipeline with Dagster guide
• New example of multi-asset conditional materialization
• New tutorial section about scheduling
• New images on the Dagster README

All Changes

1.2.3...1.2.4