How to Migrate Your Instance#

Overview#

When upgrading your Dagster version, you may also need to migrate your Dagster instance. Migrations will only be required if you are upgrading your minor version.

Here, we walk you through the migration process in a Kubernetes environment.

Walkthrough#

For simplicity, we assume that you have only installed one release of the Dagster Helm chart in your Kubernetes cluster. All commands should be run in the proper namespace.

1. Before running the migration, check the migration document for any additional steps you may need to take as part of the migration.
2. Make sure you backup the data in your PostgreSQL database!
3. Run helm upgrade with your desired Dagster chart version and Helm values.
4. Scale down the Dagster webserver and daemon deployments. We do not want any ongoing job runs to be writing to the database as the migration is happening. As we scale down the Deployments, we note each Deployment's replica count to scale each back up after the migration is complete.

# Get the names of Dagster's webserver and daemon deployments created by Helm
export WEBSERVER_DEPLOYMENT_NAME=`kubectl get deploy \
    --selector=component=dagster-webserver -o jsonpath="{.items[0].metadata.name}"`
export DAEMON_DEPLOYMENT_NAME=`kubectl get deploy \
    --selector=component=dagster-daemon -o jsonpath="{.items[0].metadata.name}"`

# Save each deployment's replica count to scale back up after migrating
export WEBSERVER_DEPLOYMENT_REPLICA_COUNT=`kubectl get deploy \
    --selector=component=dagster-webserver -o jsonpath="{.items[0].status.replicas}"`
export DAEMON_DEPLOYMENT_REPLICA_COUNT=`kubectl get deploy \
    --selector=component=dagster-daemon -o jsonpath="{.items[0].status.replicas}"`

# Scale down the Deployments
kubectl scale deploy $WEBSERVER_DEPLOYMENT_NAME --replicas=0
kubectl scale deploy $DAEMON_DEPLOYMENT_NAME --replicas=0

5. Run a Kubernetes Job with the dagster instance migrate command. You can check if the migration succeeded by inspecting the the resulting Pod of the Job.

# Run `helm list` and save your Dagster Helm release name
export HELM_DAGSTER_RELEASE_NAME=<DAGSTER_RELEASE_NAME>

# The `helm template` command must be run from the directory containing the
# `values.yaml` file you used to install the Dagster Helm chart.
#
# If needed, you can retrieve the currently applied `values.yaml` file
# from the cluster by running:
#
# `helm get values $HELM_DAGSTER_RELEASE_NAME > values.yaml`
#
helm template $HELM_DAGSTER_RELEASE_NAME dagster/dagster \
    --set "migrate.enabled=true" \
    --show-only templates/job-instance-migrate.yaml \
    --values values.yaml \
    | kubectl apply -f -

6. Scale the webserver and daemon deployments back up.

kubectl scale deploy $WEBSERVER_DEPLOYMENT_NAME --replicas=$WEBSERVER_DEPLOYMENT_REPLICA_COUNT
kubectl scale deploy $DAEMON_DEPLOYMENT_NAME --replicas=$DAEMON_DEPLOYMENT_REPLICA_COUNT