Setting Dagster Cloud environment variables using agent configuration#

Amazon ECS agents#

Using the container_context.ecs.env_vars and container_context.ecs.secrets properties, you can configure environment variables and secrets for a specific code location.

# dagster_cloud.yaml

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      ecs:
        env_vars:
          - DATABASE_NAME=testing
          - DATABASE_PASSWORD
        secrets:
          - name: "MY_API_TOKEN"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::"
          - name: "MY_PASSWORD"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::"
        secrets_tags:
          - "my_tag_name"

After you've modified dagster_cloud.yaml, redeploy the code location in Dagster Cloud to apply the changes:

Docker agents#

Using the container_context.docker.env_vars property, you can include environment variables and secrets in the Docker container associated with a specific code location. For example:

# dagster_cloud.yaml
locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      docker:
        env_vars:
          - DATABASE_NAME
          - DATABASE_USERNAME=hooli_testing

The container_context.docker.env_vars property is a list, where each item can be either KEY or KEY=VALUE. If only KEY is specified, the value will be pulled from the local environment.

After you've modified dagster_cloud.yaml, redeploy the code location in Dagster Cloud to apply the changes:

Kubernetes agents#

Using the container_context.k8s.env_vars and container_context.k8s.env_secrets properties, you can specify environment variables and secrets for a specific code location. For example:

# dagster_cloud.yaml

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      k8s:
        env_vars:
          - database_name        # value pulled from agent's environment
          - database_username=hooli_testing
        env_secrets:
          - database_password

After you've modified dagster_cloud.yaml, redeploy the code location in Dagster Cloud to apply the changes:

Amazon ECS agents#

To make environment variables accessible to a full deployment with an Amazon ECS agent, you'll need to modify the agent's CloudFormation template as follows:

1. Sign in to your AWS account.

2. Navigate to CloudFormation and open the stack for the agent.

3. Click Update.

4. Click Edit template in designer.

5. In the section that displays, click View in Designer. The AWS template designer will display.

6. In the section displaying the template YAML, locate the AgentTaskDefinition section:

   

7. In the user_code_launcher.config portion of the AgentTaskDefinition section, add the environment variables as follows:

   user_code_launcher:
     module: dagster_cloud.workspace.ecs
     class: EcsUserCodeLauncher
     config:
       cluster: ${ConfigCluster}
       subnets: [${ConfigSubnet}]
       service_discovery_namespace_id: ${ServiceDiscoveryNamespace}
       execution_role_arn: ${TaskExecutionRole.Arn}
       task_role_arn: ${AgentRole}
       log_group: ${AgentLogGroup}
       env_vars:
         - SNOWFLAKE_USERNAME=dev
         - SNOWFLAKE_PASSWORD       ## pulled from agent environment
   ' > $DAGSTER_HOME/dagster.yaml && cat $DAGSTER_HOME/dagster.yaml && dagster-cloud agent run"
   

8. When finished, click the Create Stack button:

   

9. You'll be redirected back to the Update stack wizard, where the new template will be populated. Click Next.

10. Continue to click Next until you reach the Review page.

11. Click Submit to update the stack.

Docker agents#

To make environment variables accessible to a full deployment with a Docker agent, you'll need to modify your project's dagster.yaml file.

In the user_code_launcher section, add an env_vars property as follows:

# dagster.yaml

user_code_launcher:
  module: dagster_cloud.workspace.docker
  class: DockerUserCodeLauncher
  config:
    networks:
      - dagster_cloud_agent
    env_vars:
      - SNOWFLAKE_PASSWORD # value pulled from agent's environment
      - SNOWFLAKE_USERNAME=dev

In env_vars, specify the environment variables as keys (SNOWFLAKE_PASSWORD) or key-value pairs (SNOWFLAKE_USERNAME=dev). If only KEY is provided, the value will be pulled from the agent's environment.

Kubernetes agents#

To make environment variables to a full deployment with a Kubernetes agent, you'll need to modify and upgrade the Helm chart's values.yaml.

1. In values.yaml, add or locate the workspace value.

2. Add an envVars property as follows:

   # values.yaml
   
   workspace:
     envVars:
       - SNOWFLAKE_PASSWORD # value pulled from agent's environment
       - SNOWFLAKE_USERNAME=dev
   

3. In envVars, specify the environment variables as keys (SNOWFLAKE_PASSWORD) or key-value pairs (SNOWFLAKE_USERNAME=dev). If only KEY is provided, the value will be pulled from the local (agent's) environment.

4. Upgrade the Helm chart.