Azure (dagster-azure)¶

Utilities for using Azure Storage Accounts with Dagster. This is mostly aimed at Azure Data Lake Storage Gen 2 (ADLS2) but also contains some utilities for Azure Blob Storage.

dagster_azure.adls2.adls2_resource ResourceDefinition[source]¶

Config Schema:

storage_account (dagster.StringSource):

The storage account name.

credential (selector):

The credentials with which to authenticate.

Config Schema:

sas (dagster.StringSource):

SAS token for the account.

key (dagster.StringSource):

Shared Access Key for the account.

DefaultAzureCredential (permissive dict, optional):

Default Value:

{}

Resource that gives ops access to Azure Data Lake Storage Gen2.

The underlying client is a DataLakeServiceClient.

Attach this resource definition to a JobDefinition in order to make it available to your ops.

Example

from dagster import job, op
from dagster_azure.adls2 import adls2_resource

@op(required_resource_keys={'adls2'})
def example_adls2_op(context):
    return list(context.resources.adls2.adls2_client.list_file_systems())

@job(resource_defs={"adls2": adls2_resource})
def my_job():
    example_adls2_op()

Note that your ops must also declare that they require this resource with required_resource_keys, or it will not be initialized for the execution of their compute functions.

You may pass credentials to this resource using either a SAS token, a key or by passing the DefaultAzureCredential object.

resources:
  adls2:
    config:
      storage_account: my_storage_account
      # str: The storage account name.
      credential:
        sas: my_sas_token
        # str: the SAS token for the account.
        key:
          env: AZURE_DATA_LAKE_STORAGE_KEY
        # str: The shared access key for the account.
        DefaultAzureCredential: {}
        # dict: The keyword arguments used for DefaultAzureCredential
        # or leave the object empty for no arguments
        DefaultAzureCredential:
            exclude_environment_credential: true

class dagster_azure.adls2.FakeADLS2Resource(*, account_name, storage_account=None)[source]¶

Stateful mock of an ADLS2Resource for testing.

Wraps a mock.MagicMock. Containers are implemented using an in-memory dict.

class dagster_azure.blob.AzureBlobComputeLogManager(storage_account, container, secret_key=None, local_dir=None, inst_data=None, prefix='dagster', upload_interval=None, default_azure_credential=None)[source]¶

Logs op compute function stdout and stderr to Azure Blob Storage.

This is also compatible with Azure Data Lake Storage.

Users should not instantiate this class directly. Instead, use a YAML block in dagster.yaml such as the following:

compute_logs:
  module: dagster_azure.blob.compute_log_manager
  class: AzureBlobComputeLogManager
  config:
    storage_account: my-storage-account
    container: my-container
    credential: sas-token-or-secret-key
    default_azure_credential:
      exclude_environment_credential: true
    prefix: "dagster-test-"
    local_dir: "/tmp/cool"
    upload_interval: 30

Parameters:

• storage_account (str) – The storage account name to which to log.

• container (str) – The container (or ADLS2 filesystem) to which to log.

• secret_key (Optional[str]) – Secret key for the storage account. SAS tokens are not supported because we need a secret key to generate a SAS token for a download URL.

• default_azure_credential (Optional[dict]) – Use and configure DefaultAzureCredential. Cannot be used with sas token or secret key config.

• local_dir (Optional[str]) – Path to the local directory in which to stage logs. Default: dagster._seven.get_system_temp_directory().

• prefix (Optional[str]) – Prefix for the log file keys.

• upload_interval – (Optional[int]): Interval in seconds to upload partial log files blob storage. By default, will only upload when the capture is complete.

• inst_data (Optional[ConfigurableClassData]) – Serializable representation of the compute log manager when newed up from config.

dagster_azure.adls2.adls2_pickle_io_manager IOManagerDefinition[source]¶

Config Schema:

adls2 (Union[Any, None], optional):

adls2_file_system (dagster.StringSource):

ADLS Gen2 file system name.

adls2_prefix (dagster.StringSource, optional):

ADLS Gen2 file system prefix to write to.

Default Value: ‘dagster’

Persistent IO manager using Azure Data Lake Storage Gen2 for storage.

Serializes objects via pickling. Suitable for objects storage for distributed executors, so long as each execution node has network connectivity and credentials for ADLS and the backing container.

Assigns each op output to a unique filepath containing run ID, step key, and output name. Assigns each asset to a single filesystem path, at “<base_dir>/<asset_key>”. If the asset key has multiple components, the final component is used as the name of the file, and the preceding components as parent directories under the base_dir.

Subsequent materializations of an asset will overwrite previous materializations of that asset. With a base directory of “/my/base/path”, an asset with key AssetKey([“one”, “two”, “three”]) would be stored in a file called “three” in a directory with path “/my/base/path/one/two/”.

Example usage:

1. Attach this IO manager to a set of assets.

from dagster import Definitions, asset
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource

@asset
def asset1():
    # create df ...
    return df

@asset
def asset2(asset1):
    return df[:5]

defs = Definitions(
    assets=[asset1, asset2],
    resources={
        "io_manager": adls2_pickle_io_manager.configured(
            {"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
        ),
        "adls2": adls2_resource,
    },
)

2. Attach this IO manager to your job to make it available to your ops.

from dagster import job
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource

@job(
    resource_defs={
        "io_manager": adls2_pickle_io_manager.configured(
            {"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
        ),
        "adls2": adls2_resource,
    },
)
def my_job():
    ...