Changelog#

0.9.1#

Bugfixes

Fixes an issue in the dagster-k8s-celery executor when executing solid subsets

Breaking Changes

Deprecated the IntermediateStore API. IntermediateStorage now wraps an ObjectStore, and TypeStoragePlugin now accepts an IntermediateStorage instance instead of an IntermediateStore instance. (Noe that IntermediateStore and IntermediateStorage are both internal APIs that are used in some non-core libraries).

Breaking Changes

The dagit key is no longer part of the instance configuration schema and must be removed from dagster.yaml files before they can be used.
-d can no longer be used as a command-line argument to specify a mode. Use --mode instead.
Use --preset instead of --preset-name to specify a preset to the pipeline launch command.
We have removed the config argument to the ConfigMapping, @composite_solid, @solid, SolidDefinition, @executor, ExecutorDefinition, @logger, LoggerDefinition, @resource, and ResourceDefinition APIs, which we deprecated in 0.8.0. Use config_schema instead.

New

Python 3.8 is now fully supported.
-d or --working-directory can be used to specify a working directory in any command that takes in a -f or --python_file argument.
Removed the deprecation of create_dagster_pandas_dataframe_type. This is the currently supported API for custom pandas data frame type creation.
Removed gevent dependency from dagster
New configured API for predefining configuration for various definitions: https://docs.dagster.io/overview/configuration/#configured
Added hooks to enable success and failure handling policies on pipelines. This enables users to set up policies on all solids within a pipeline or on a per solid basis. Example usage can be found here
New instance level view of Scheduler and running schedules
dagster-graphql is now only required in dagit images.

Breaking Changes

AssetMaterializations no longer accepts a dagster_type argument. This reverts the change billed as "AssetMaterializations can now have type information attached as metadata." in the previous release.

New

Added new GCS and Azure file manager resources
AssetMaterializations can now have type information attached as metadata. See the materializations tutorial for more
Added verification for resource arguments (previously only validated at runtime)

Bugfixes

Fixed bug with order-dependent python module resolution seen with some packages (e.g. numpy)
Fixed bug where Airflow's context['ts'] was not passed properly
Fixed a bug in celery-k8s when using task_acks_late: true that resulted in a 409 Conflict error from Kubernetes. The creation of a Kubernetes Job will now be aborted if another Job with the same name exists
Fixed a bug with composite solid output results when solids are skipped
Hide the re-execution button in Dagit when the pipeline is not re-executable in the currently loaded repository

Docs

New

CeleryK8sRunLauncher supports termination of pipeline runs. This can be accessed via the “Terminate” button in Dagit’s Pipeline Run view or via “Cancel” in Dagit’s All Runs page. This will terminate the run master K8s Job along with all running step job K8s Jobs; steps that are still in the Celery queue will not create K8s Jobs. The pipeline and all impacted steps will be marked as failed. We recommend implementing resources as context managers and we will execute the finally block upon termination.
K8sRunLauncher supports termination of pipeline runs.
AssetMaterialization events display the asset key in the Runs view.
Added a new "Actions" button in Dagit to allow to cancel or delete mulitple runs.

Bugfixes

Fixed an issue where DagsterInstance was leaving database connections open due to not being garbage collected.
Fixed an issue with fan-in inputs skipping when upstream solids have skipped.
Fixed an issue with getting results from composites with skippable outputs in python API.
Fixed an issue where using Enum in resource config schemas resulted in an error.