You are viewing an outdated version of the documentation.

This documentation is for an older version (1.4.7) of Dagster. You can view the version of this page from our latest release below.

Code locations#

Before the introduction of the DefinitionsAPI, definitions were grouped into repositories, and there could be many repostories in a particular code location. Refer to the Repositories documentation for info on this previous API and mental model.

Code locations

A code location is a collection of Dagster definitions loadable and accessible by Dagster's tools, such as the CLI, UI, and Dagster Cloud. A code location comprises:

  • A reference to a Python module that has an instance of Definitions in a top-level variable
  • A Python environment that can successfully load that module

Definitions within a code location have a common namespace and must have unique names. This allows them to be grouped and organized by code location in tools.

A single deployment can have one or multiple code locations.

Code locations are loaded in a different process and communicate with Dagster system processes over an RPC mechanism. This architecture provides several advantages:

  • When there is an update to user code, the Dagster webserver/UI can pick up the change without a restart.
  • You can use multiple code locations to organize jobs, but still work on all of your code locations using a single instance of the webserver/UI.
  • The Dagster webserver process can run in a separate Python environment from user code so job dependencies don't need to be installed into the webserver environment.
  • Each code location can be sourced from a separate Python environment, so teams can manage their dependencies (or even their Python versions) separately.

Relevant APIs#

NameDescription
DefinitionsThe object that contains all the definitions defined within a code location. Definitions include assets, jobs, resources, schedules, and sensors.

Defining code locations#

To define a code location, create a top-level variable that contains a Definitions object in a Python module. For example:

# my_file.py

defs = Definitions(
    assets=[dbt_customers_asset, dbt_orders_asset],
    schedules=[bi_weekly_schedule],
    sensors=[new_data_sensor],
    resources=dbt_resource
)

Definitions can be included in a Python file like my_file.py or a Python module. If using the latter, the Definitions object should be defined in the module's top-level __init__.py file.


Deploying and loading code locations#

Local development#

Refer to the Running Dagster locally guide for more info about local development, including how to configure your local instance.

Dagster can load a file directly as a code location. In the following example, we used the -f argument to supply the name of the file:

dagster dev -f my_file.py

This command loads the definitions in my_file.py as a code location in the current Python environment.

You can also include multiple files at a time, where each file will be loaded as a code location:

dagster dev -f my_file.py -f my_second_file.py

Cloud deployment#

The dagster_cloud.yaml file is used to create and deploy code locations for Cloud deployments. Each code location entry in this file has a code_source property, which is used to specify how a code location is sourced. Code locations can be sourced from a Python file or module:

To load a code location from a Python file, use the python_file property in your dagster_cloud.yaml:

# dagster_cloud.yaml

locations:
  - location_name: my-code-location
    code_source:
      python_file: my_file.py

Open source deployment#

The workspace.yaml file is used to load code locations for open source (OSS) deployments. This file specifies how to load a collection of code locations and is typically used in advanced use cases. Refer to the Workspace files page for more info.


Definitions versus repositories#

If you used @repository in previous Dagster versions, you might be interested in how Definitions and repositories differ. Check out the following table for a high-level comparison:

Definitions (Recommended)Repositories
Minimum Dagster version1.1.70.6
Description
  • Created by using the Definitions object assigned to a top-level variable
  • One Definitions object allowed per code location
ArgumentsEnforced typing and namingNo enforced typing and naming
Resources
  • resources argument can accept definitions and raw objects
  • Top-level resources are automatically bound to all assets
Resources are manually bound to assets (with_resources)
Multiple Python environmentsSupported for OSS deployments (viaworkspace.yaml)Supported

Troubleshooting#

ErrorDescription and resolution
Cannot have more than one Definitions object defined at module scopeDagster found multiple Definitions objects in a single Python module. Only one Definitions object may be in a single code location.