Running multiple agents

(Experimental)

#

Each Dagster Cloud full deployment (e.g., prod) needs to have at least one agent running. A single agent is adequate for many use cases, but you may want to run multiple agents to provide redundancy if a single agent goes down.

It's recommended to only use multiple agents of the same type (e.g. multiple Kubernetes agents). Submitting runs to a specific agent isn't currently supported.

Multiple agents in the same environment#

If you want to run multiple agents in the same environment (e.g., multiple Kubernetes agents in the same namespace), you can set the number of replicas in the configuration for your particular agent type:

In Docker#

In Docker, you can set the number of replicas for a service in the docker-compose.yaml file if the deployment mode is set to replicated (which is the default):

services:
  dagster-cloud-agent:
    ...
    deploy:
      mode: replicated
      replicas: 2

In Kubernetes#

In Kubernetes, the number of replicas is set in the Helm chart. You can set the number of replicas in the Helm command:

helm upgrade \
    ...
    --set replicas=2

or if using a values.yaml file:

dagsterCloudAgent:
  ...
  replicas: 2

In Amazon ECS#

In Amazon ECS, the number of replicas can be set via the CloudFormation template:

DagsterCloudAgent:
  Type: AWS::ECS::Service
  Properties:
    ...
    DesiredCount: 2

If using the CloudFormation template provided by Dagster, the number of replicas can be set via the NumReplicas parameter in the Amazon Web Services (AWS) UI.

Multiple agents in different environments#

If you want to run multiple agents in an environment where each agent can not access the others' resources (for example, multiple kubernetes namespaces or different clusters), you need to enable the isolated_agents option. This is supported for all agent types.

In Docker#

Add the following to the dagster.yaml file:

isolated_agents:
  enabled: true

dagster_cloud_api:
  # <your other config>
  agent_label: "My agent" # optional

In Kubernetes#

Add the following options to your Helm command:

helm upgrade \
    ...
    --set agentReplicas.enabled=true \
    --set dagsterCloud.agentLabel="My agent" # optional, only supported on 0.13.14 and later

Or if you're using a values.yaml file:

isolatedAgents:
  enabled: true

dagsterCloud:
  agentLabel: "My agent" # optional, only supported on 0.13.14 and later

In Amazon ECS#

The isolated_agents option can be set as per-deployment configuration on the dagster.yaml file used by your agent. See the ECS configuration reference guide for more information.