Getting started with Dagster Cloud#

With GitHub

1. Click Select to select a template.
2. Sign in to your GitHub account, if prompted.
3. In Dagster Cloud, define the following:
   • Git scope - Select the organization or account to create the new repository in.
   • Repository Name - Enter a name for the new repository.
   • Check the Make git repository private box to make the repository private.
4. When finished, click Clone and deploy.

When finished, continue to the next step.

With Gitlab

1. Click Select to select a template.
2. Sign in to your Gitlab account, if prompted.
3. In Dagster Cloud, define the following:
   • Namespace - Select the group or account to create the new project in.
   • Project - Enter a name for the new project.
   • Check the Make git project private box to make the project private.
4. When finished, click Clone and deploy.

When finished, continue to the next step.

To import existing Dagster code, you can use Dagster's GitHub / Gitlab app or the dagster-cloud CLI.

Using the GitHub integration

Using the GitHub integration to import an existing GitHub repository also sets up CI/CD for you.

Before you get started, note that the repository must have a dagster_cloud.yaml file in order for Dagster Cloud to deploy it. This file defines the code locations in your Dagster code.

If this file doesn't currently exist, create it in the root of your repository with the following code:

# dagster_cloud.yaml

locations:
  - location_name: my_location_name # the name of the code location
    code_source:
      package_name: hackernews # the name of the python package associated with the code location

After you've committed the file to the repository, come back to Dagster Cloud to complete the import process:

1. Click the Import an existing project tab.
2. Sign in to your GitHub account, if prompted.
3. In Dagster Cloud, define the following:
   • Git scope - Select the organization or account that the repository is in.
   • Repository - Select the repository.
4. Click Deploy.

When finished, continue to the next step.

Using the Gitlab integration

Using the Gitlab integration to import an existing Gitlab project also sets up CI/CD for you.

Before you get started, note that the project must have a dagster_cloud.yaml file in order for Dagster Cloud to deploy it. This file defines the code locations in your Dagster code.

If this file doesn't currently exist, create it in the root of your repository with the following code:

# dagster_cloud.yaml

locations:
  - location_name: my_location_name # the name of the code location
    code_source:
      package_name: hackernews # the name of the python package associated with the code location

After you've committed the file to the project, come back to Dagster Cloud to complete the import process:

1. Click the Import an existing project tab.
2. Sign in to your Gitlab account, if prompted.
3. In Dagster Cloud, define the following:
   • Namespace - Select the group or account that the project is in.
   • Project - Select the project.
4. Click Deploy.

When finished, continue to the next step.

Using the dagster-cloud CLI

To complete this step using the CLI, you can use your own Dagster code or the Dagster starter kit. The starter kit is a template with everything you need to get started using Serverless in Dagster Cloud, including CI/CD configuration and the required dagster_cloud.yaml file.

• If using the template, clone the repository to your local environment.

• If not using the template, add dagster-cloud as a dependency in setup.py. [Click here for an example](https://github.com/dagster-io/quickstart-etl/blob/main/setup.py. This is already done for you if using the starter kit.

After you've finished setting up your local project, move on to deploying using the CLI:

1. To install the dagster-cloud CLI, run:

   pip install dagster-cloud
   

2. Next, you'll need to authenticate to Dagster Cloud. Run the following command and follow the prompts to log in:

   dagster-cloud config setup
   

3. After you've successfully authenticated, run the following to deploy your code to Dagster Cloud:

   dagster-cloud serverless deploy \
         --location-name "<LOCATION_NAME>" \       # name of the code loation
         --package-name "<PACKAGE_NAME>" \           # name of the Python package associated with the code location
   

When finished, continue to the next step.

To set up continuous integration using GitHub Actions, you can use your own Dagster code or the Dagster Cloud Hybrid Quickstart. The quickstart is a template with everything you need to get started using Hybrid deployment in Dagster Cloud.

• If using the template, clone the repository.

• If not using the template, copy the GitHub workflow files (.github/workflows) from the Hybrid quickstart repository and add them to your repository. This is already done for you if using the quickstart.

Configure the GitHub workflow YAML file as described below. The GitHub workflow deploys your code to Dagster Cloud using these steps:

1. Initialize - check out your code and validate dagster_cloud.yaml.

2. Docker image push - build a Docker image from your code and upload it to your container registry.

3. Deploy to Dagster Cloud - update code locations in Dagster Cloud to use the new Docker image.

To configure the workflow, follow these steps:

1. In the repository, set the DAGSTER_CLOUD_API_TOKEN GitHub action secret. This is the Dagster Cloud agent token from the previous section. Refer to the agent tokens documentation for more info.

   Refer to the GitHub docs for more info about GitHub Action secrets.

2. In your dagster-cloud-deploy.yml, set the DAGSTER_CLOUD_ORGANIZATION environment variable to your Dagster Cloud organization name.

3. In your dagster-cloud-deploy.yml, uncomment the step that is relevant to your Docker container registry. For example, if using DockerHub, you'd uncomment the DockerHub step in these files. Ensure you have set up the relevant secrets for building and uploading your Docker images.

After making the above changes and commiting the workflow file, the CI process should be triggered to deploy your GitHub repository to Dagster Cloud. During the deployment, the agent will attempt to load your code and update the metadata in Dagster Cloud. Once finished, you should see the GitHub Action complete successfully and also be able to see the code location under the Deployment tag in Dagster Cloud.

When finished, continue to the next step.

For continuous integration using a CI/CD provider other than GitHub, your system should use the dagster-cloud ci subcommand to deploy code locations to Dagster Cloud.

Ensure that you have created a dagster_cloud.yaml file as described in the quickstart. For detailed documentation about this file see the dagster_cloud.yaml reference. The following steps are typically implemented in the CI/CD workflow:

1. Set the build environment variables. Note that all variables are required:

   • DAGSTER_CLOUD_ORGANIZATION - The name of your organization in Dagster Cloud.
   • DAGSTER_CLOUD_API_TOKEN - A Dagster Cloud API token. Note: This is a sensitive value and should be stored as a CI/CD secret, if possible.
   • DAGSTER_BUILD_STATEDIR - A path to a blank or non-existent temporary directory on the build machine. This directory is used to store local state during the build.

2. Run the configuration check:

   dagster-cloud ci check --project-dir=.
   

   This is an optional step but useful to validate the contents of your dagster_cloud.yaml and connection to Dagster Cloud.

3. Initialize the build session:

   dagster-cloud ci init --project-dir=.
   

   This reads the dagster_cloud.yaml configuration and initializes the DAGSTER_BUILD_STATEDIR.

4. Build and upload Docker images for your code locations.

   The Docker image should contain a Python environment with dagster, dagster-cloud, and your code. For reference, see the example Dockerfile in our quickstart repository. The example uses pip install . to install the code including the dependencies specified in setup.py.

   It is a good idea to use a unique image tag for each Docker build. You can build one image per code location or a shared image for multiple code locations. As an example image tag, you can use the git commit SHA:

   export IMAGE_TAG=`git log --format=format:%H -n 1`
   

   Use this tag to build and upload your Docker image, for example:

   docker build . -t ghcr.io/org/dagster-cloud-image:$IMAGE_TAG
   docker push ghcr.io/org/dagster-cloud-image:$IMAGE_TAG
   

   The upload step is specific to your Docker container registry and will require authentication. The only requirement is that the registry you upload to must match the registry specified in dagster_cloud.yaml.

5. Update the build session with the Docker image tag. For each code location you want to deploy, run the following command passing the IMAGE_TAG used in the previous step:

   dagster-cloud ci set-build-output --location-name=code-location-a --image-tag=IMAGE_TAG
   

   This command does not deploy the code location but just updates the local state in DAGSTER_BUILD_STATEDIR.

6. Deploy to Dagster Cloud:

   dagster-cloud ci deploy
   

   This command updates the code locations in Dagster Cloud. Once this finishes successfully, you should be able to see the code locations under the Deployments tab in Dagster Cloud.

Note: Creating Branch Deployments using the CLI requires some additional steps. Refer to the Branch Deployments with the dagster-cloud CLI guide for more info.

When finished, continue to the next step.