Changelog#

0.6.0 "Impossible Princess"#

New

This major release includes features for scheduling, operating, and executing pipelines that elevate Dagit and dagster from a local development tool to a deployable service.

  • DagsterInstance introduced as centralized system to control run, event, compute log, and local intermediates storage.
  • A Scheduler abstraction has been introduced along side an initial implementation of SystemCronScheduler in dagster-cron.
  • dagster-aws has been extended with a CLI for deploying dagster to AWS. This can spin up a Dagit node and all the supporting infrastructure—security group, RDS PostgreSQL instance, etc.—without having to touch the AWS console, and for deploying your code to that instance.
  • Dagit
    • Runs: a completely overhauled Runs history page. Includes the ability to Retry, Cancel, and Delete pipeline runs from the new runs page.
    • Scheduler: a page for viewing and interacting with schedules.
    • Compute Logs: stdout and stderr are now viewable on a per execution step basis in each run. This is available in real time for currently executing runs and for historical runs.
    • A Reload button in the top right in Dagit restarts the web-server process and updates the UI to reflect repo changes, including DAG structure, solid names, type names, etc. This replaces the previous file system watching behavior.

Breaking Changes

  • --log and --log-dir no longer supported as CLI args. Existing runs and events stored via these flags are no longer compatible with current storage.
  • raise_on_error moved from in process executor config to argument to arguments in python API methods such as execute_pipeline

0.5.9#

  • Fixes an issue using custom types for fan-in dependencies with intermediate storage.

0.5.8#

  • Fixes an issue running some Dagstermill notebooks on Windows.
  • Fixes a transitive dependency issue with Airflow.
  • Bugfixes, performance improvements, and better documentation.

0.5.7#

  • Fixed an issue with specifying composite output mappings (#1674)
  • Added support for specifying Dask worker resources (#1679)
  • Fixed an issue with launching Dagit on Windows

0.5.6#

  • Execution details are now configurable. The new top-level ExecutorDefinition and @executor APIs are used to define in-process, multiprocess, and Dask executors, and may be used by users to define new executors. Like loggers and storage, executors may be added to a ModeDefinition and may be selected and configured through the execution field in the environment dict or YAML, including through Dagit. Executors may no longer be configured through the RunConfig.
  • The API of dagster-dask has changed. Pipelines are now executed on Dask using the ordinary execute_pipeline API, and the Dask executor is configured through the environment. (See the dagster-dask README for details.)
  • Added the PresetDefinition.from_files API for constructing a preset from a list of environment files (replacing the old usage of this class). PresetDefinition may now be directly instantiated with an environment dict.
  • Added a prototype integration with dbt.
  • Added a prototype integration with Great Expectations.
  • Added a prototype integration with Papertrail.
  • Added the dagster-bash library.
  • Added the dagster-ssh library.
  • Added the dagster-sftp library.
  • Loosened the PyYAML compatibility requirement.
  • The dagster CLI no longer takes a --raise-on-error or --no-raise-on-error flag. Set this option in executor config.
  • Added a MarkdownMetadataEntryData class, so events yielded from client code may now render markdown in their metadata.
  • Bug fixes, documentation improvements, and improvements to error display.