Changelog#

0.6.5#

Breaking

  • The interface for type checks has changed. Previously the type_check_fn on a custom type was required to return None (=passed) or else raise Failure (=failed). Now, a type_check_fn may return True/False to indicate success/failure in the ordinary case, or else return a TypeCheck. The newsuccess field on TypeCheck now indicates success/failure. This obviates the need for the typecheck_metadata_fn, which has been removed.
  • Executions of individual composite solids (e.g. in test) now produce a CompositeSolidExecutionResult rather than a SolidExecutionResult.
  • dagster.core.storage.sqlite_run_storage.SqliteRunStorage has moved to dagster.core.storage.runs.SqliteRunStorage. Any persisted dagster.yaml files should be updated with the new classpath.
  • is_secret has been removed from Field. It was not being used to any effect.
  • The environmentType and configTypes fields have been removed from the dagster-graphql Pipeline type. The configDefinition field on SolidDefinition has been renamed to configField.

Bugfix

  • PresetDefinition.from_files is now guaranteed to give identical results across all Python minor versions.
  • Nested composite solids with no config, but with config mapping functions, now behave as expected.
  • The dagster-airflow DagsterKubernetesPodOperator has been fixed.
  • Dagit is more robust to changes in repositories.
  • Improvements to Dagit interface.

New

  • dagster_pyspark now supports remote execution on EMR with the @pyspark_solid decorator.

Nits

  • Documentation has been improved.
  • The top level config field features in the dagster.yaml will no longer have any effect.
  • Third-party dependencies have been relaxed to reduce the risk of version conflicts.

0.6.4#

  • Scheduler errors are now visible in Dagit
  • Run termination button no longer persists past execution completion
  • Fixes run termination for multiprocess execution
  • Fixes run termination on Windows
  • dagit no longer prematurely returns control to terminal on Windows
  • raise_on_error is now available on the execute_solid test utility
  • check_dagster_type added as a utility to help test type checks on custom types
  • Improved support in the type system for Set and Tuple types
  • Allow composite solids with config mapping to expose an empty config schema
  • Simplified graphql API arguments to single-step re-execution to use retryRunId, stepKeys execution parameters instead of a reexecutionConfig input object
  • Fixes missing step-level stdout/stderr from dagster CLI

0.6.3#

  • Adds a type_check parameter to PythonObjectType, as_dagster_type, and @as_dagster_type to enable custom type checks in place of default isinstance checks. See documentation here: https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/types.html#custom-type-checks

  • Improved the type inference experience by automatically wrapping bare python types as dagster types.

  • Reworked our tutorial (now with more compelling/scary breakfast cereal examples) and public API documentation. See the new tutorial here: https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/index.html

  • New solids explorer in Dagit allows you to browse and search for solids used across the repository.

  • Enabled solid dependency selection in the Dagit search filter.

    • To select a solid and its upstream dependencies, search +{solid_name}.
    • To select a solid and its downstream dependents, search {solid_name}+.
    • For both search +{solid_name}+.

  • Added a terminate button in Dagit to terminate an active run.

  • Added an --output flag to dagster-graphql CLI.

  • Added confirmation step for dagster run wipe and dagster schedule wipe commands (Thanks @shahvineet98).

  • Fixed a wrong title in the dagster-snowflake library README (Thanks @Step2Web).

0.6.2#

  • Changed composition functions @pipeline and @composite_solid to automatically give solids aliases with an incrementing integer suffix when there are conflicts. This removes to the need to manually alias solid definitions that are used multiple times.
  • Add dagster schedule wipe command to delete all schedules and remove all schedule cron jobs
  • execute_solid test util now works on composite solids.
  • Docs and example improvements: https://dagster.readthedocs.io/
  • Added --remote flag to dagster-graphql for querying remote Dagit servers.
  • Fixed issue with duplicate run tag autocomplete suggestions in Dagit (#1839)
  • Fixed Windows 10 / py3.6+ bug causing pipeline execution failures

0.6.1#

  • Fixed an issue where Dagster public images tagged latest on Docker Hub were erroneously published with an older version of Dagster (#1814)
  • Fixed an issue where the most recent scheduled run was not displayed in Dagit (#1815)
  • Fixed a bug with the dagster schedule start --start-all command (#1812)
  • Added a new scheduler command to restart a schedule: dagster schedule restart. Also added a flag to restart all running schedules: dagster schedule restart --restart-all-running.