You are viewing an outdated version of the documentation.

This documentation is for an older version (1.4.7) of Dagster. You can view the version of this page from our latest release below.

View Latest Documentation

Using Dagster with Airflow#

You can find the code for this example on Github

The dagster-airflow package provides interoperability between Dagster and Airflow. The main scenarios for using the Dagster Airflow integration are:

  • You want to do a lift-and-shift migration of all your existing Airflow DAGs into Dagster Jobs/SDAs
  • You want to trigger Dagster job runs from Airflow

This integration is designed to help support users who have existing Airflow usage and are looking to explore using Dagster.

Airflow vs Dagster concept map#

While Airflow and Dagster have some significant differences, there are many concepts that overlap. To ease the transition, we recommend using this cheatsheet to understand how Airflow concepts map to Dagster.

Airflow conceptDagster conceptNotes
DAG Job
Task Op
OperatorNoneDagster uses normal Python functions instead of framework-specific operator classes. For off-the-shelf functionality with third-party tools, Dagster provides integration libraries.
Scheduler Scheduler
Executor Executor
DagBag Code LocationsMultiple isolated code locations with different system and Python dependencies can exist within the same Dagster instance.
Instance Instance
SubDAGs / TaskGroups Graphs, Tags and grouped assetsDagster provides rich, searchable metadata and tagging support well beyond what’s offered by Airflow.
Hooks ResourcesDagster resources contain a superset of the functionality of hooks and have much stronger composition guarantees.
Pools Run Coordinator
XComs IO ManagerI/O managers are more powerful than XComs and allow the passing large datasets between jobs.
Trigger LaunchpadTriggering and configuring ad-hoc runs is easier in Dagster which allows them to be initiated through the Dagster UI, the GraphQL API, or the CLI.
Sensor Sensor
DAG runJob run
Plugins/Providers Integrations
Datasets Software-defined assets (SDAs)SDAs are more powerful and mature than datasets and include support for things like partitioning.
Connections/Variables Run config, Configured API and environment variables (Dagster Cloud only)