Using Branch Deployments with Gitlab CI/CD#

This guide is applicable to Dagster Cloud.

In this guide, we'll walk you through setting up Branch Deployments with Gitlab CI/CD.

Using this approach to branch deployments may be a good fit if:

You use Gitlab for version control
You want Dagster to fully automate Branch Deployments

If you don't use Gitlab for version control or want full control over your setup, check out the Branch deployments with the dagster-cloud CLI guide.

Prerequisites#

To complete the steps in this guide, you'll need:

Organization Admin permissions in Dagster Cloud
The ability to run a new agent in your infrastructure. This isn't required if you're using Serverless deployment.
The ability to configure Gitlab CI/CD for your project. This isn't required if you used the Dagster Cloud Gitlab app to connect your project as a code location.

Step 1: Generate a Dagster Cloud agent token#

In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.

Sign in to your Dagster Cloud instance.
Click the user menu (your icon) > Cloud settings.
In the Cloud settings page, click the Tokens tab.
Click the + Create agent token button.
After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.

Step 2: Create and configure an agent#

If using Serverless deployment, this step isn't required.

While you can use your existing production agent, we recommend creating a dedicated branch deployment agent. This ensures that your production instance isn't negatively impacted by the workload associated with branch deployments.

AMAZON ECS AGENTS

Deploy an ECS agent to serve your branch deployments. Follow the ECS agent setup guide, making sure to set the Enable Branch Deployments parameter if using the CloudFormation template. If you are running an existing agent, follow the upgrade guide to ensure your template is up-to-date. Then, turn on the Enable Branch Deployments parameter.
Create a private Amazon Elastic Registry (ECR) repository. Refer to the AWS ECR documentation for instructions.
After the repository has been created, navigate back to the list of ECR repositories.
In the list, locate the repository and its URI:
Create an IAM user. This user must:
- Have push access to the ECR repository, and
- Have programmatic access to AWS using an access key
After the user is created, save the Access key ID and Secret access key values shown on the confirmation page:

DOCKER AGENTS

Set up a new Docker agent. Refer to the Docker agent setup guide for instructions.

After the agent is set up, modify the dagster.yaml file as follows:

Set the dagster_cloud_api.branch_deployments field to true
Remove any deployment field(s)

For example:


# dagster.yaml

instance_class:
  module: dagster_cloud.instance
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token: <YOUR_AGENT_TOKEN>
  branch_deployments: true ## true enables branch deployments

user_code_launcher:
  module: dagster_cloud.workspace.docker
  class: DockerUserCodeLauncher
  config:
    networks:
      - dagster_cloud_agent
    server_ttl:
      enabled: true
      ttl_seconds: 7200 #2 hours

KUBERNETES AGENTS

Set up a new Kubernetes agent. Refer to the Kubernetes agent setup guide for instructions.

After the agent is set up, modify your Helm values file to include the following:


dagsterCloud:
  branchDeployments: true
---
workspace:
  serverTTL:
    enabled: true
    ttlSeconds: 7200

Step 3: Add Gitlab CI/CD script to your project#

If you used the Gitlab app to add the project as a code location, this step isn't required. Skip to the next step.

In this step, you'll add the required Gitlab CI config file to your Gitlab project. The file can be found in our CI/CD workflow repository or Serverless quickstart repository, depending on the agent you are using.

Copy the following files to your project (the linked files are shown for Hybrid repos, there are equivalents for Serverless repos).

In the next step, you'll modify these files to work with your Dagster Cloud setup.

Step 4: Configure the Gitlab project#

In this section, you'll:

Add the agent registry to dagster_cloud.yaml
Configure Gitlab CI/CD variables
Update Gitlab CI/CD script
Verify CI/CD pipeline runs

Step 4.1: Add the agent registry to dagster_cloud.yaml#

If you're using Serverless deployment, this step isn't required. Skip to the next step.

In the dagster_cloud.yaml file, replace build.registry with the registry used by the agent you created in Step 2.

For example:


# dagster_cloud.yaml

locations:
  - location_name: example_location
    code_source:
      python_file: repo.py
    build:
      directory: ./example_location
      registry: 764506304434.dkr.ecr.us-east-1.amazonaws.com/branch-deployment-agent

Step 4.2: Configure Gitlab CI/CD variables#

If you used the Gitlab app to add the project as a code location, this step isn't required.

Want to use secrets in your Dagster code? Check out the Dagster Cloud environment variables and secrets guide for more info.

In your project, click the Settings tab.
In the CI/CD section of the sidebar, expand Variables.
Click Add variable.
In the Key field, enter the name of the variable. For example, API_TOKEN
In the Value field, paste the value of the variable.
Click Add variable.

For Hybrid deployments, repeat steps 3-6 for each of the secrets required for your registry type:

DAGSTER_CLOUD_API_TOKEN - The Dagster Cloud agent token you created in Step 1
DAGSTER_CLOUD_URL - Your Dagster Cloud base URL (e.g. https://my_org.dagster.cloud)

Step 4.3: Update Gitlab CI/CD script#

If you're using Serverless deployment, this step isn't required. Skip to the next step.

In this step, you'll update the Gitlab CI/CD config to set up Docker registry access.

In the .gitlab-ci.yml file, uncomment the step associated with your registry. For example, for the Gitlab container registry, you'd uncomment the following portion of the .gitlab-ci.yml file:


build-image:
  ...
  before_script:
    # For Gitlab Container Registry
    - echo $CI_JOB_TOKEN | docker login --username $CI_REGISTRY_USER --password-stdin $REGISTRY_URL

Save and commit the files to the project.

Step 4.4: Verify pipeline runs#

The last step is to verify that the Gitlab pipeline runs successfully.

On the project page, click the CI/CD tab.
In the list of pipelines, locate the latest branch deployment run. For example:

If the run finished successfully, that's it!

Step 5: Access the branch deployment#

Now that Branch Deployments are configured, you can check out the preview in Dagster Cloud, by accessing the branch deployment from the deployment switcher:

Highlighted branch deployment in the Dagster Cloud deployment switcher

You are viewing an outdated version of the documentation.