Fivetran (dagster-fivetran)¶

Ops¶

dagster_fivetran.fivetran_sync_op = <dagster._core.definitions.op_definition.OpDefinition object>[source]¶

Config Schema:

connector_id (dagster.StringSource):

The Fivetran Connector ID that this op will sync. You can retrieve this value from the “Setup” tab of a given connector in the Fivetran UI.

poll_interval (Float, optional):

The time (in seconds) that will be waited between successive polls.

Default Value: 10

poll_timeout (Union[Float, None], optional):

The maximum time that will waited before this operation is timed out. By default, this will never time out.

yield_materializations (dagster.BoolSource, optional):

If True, materializations corresponding to the results of the Fivetran sync will be yielded when the op executes.

Default Value: True

asset_key_prefix (List[dagster.StringSource], optional):

If provided and yield_materializations is True, these components will be used to prefix the generated asset keys.

Default Value: [‘fivetran’]

Executes a Fivetran sync for a given connector_id, and polls until that sync completes, raising an error if it is unsuccessful. It outputs a FivetranOutput which contains the details of the Fivetran connector after the sync successfully completes, as well as details about which tables the sync updates.

It requires the use of the fivetran_resource, which allows it to communicate with the Fivetran API.

Examples

from dagster import job
from dagster_fivetran import fivetran_resource, fivetran_sync_op

my_fivetran_resource = fivetran_resource.configured(
    {
        "api_key": {"env": "FIVETRAN_API_KEY"},
        "api_secret": {"env": "FIVETRAN_API_SECRET"},
    }
)

sync_foobar = fivetran_sync_op.configured({"connector_id": "foobar"}, name="sync_foobar")

@job(resource_defs={"fivetran": my_fivetran_resource})
def my_simple_fivetran_job():
    sync_foobar()

@job(resource_defs={"fivetran": my_fivetran_resource})
def my_composed_fivetran_job():
    final_foobar_state = sync_foobar(start_after=some_op())
    other_op(final_foobar_state)

Resources¶

dagster_fivetran.fivetran_resource ResourceDefinition[source]¶

Config Schema:

api_key (dagster.StringSource):

The Fivetran API key to use for this resource.

api_secret (dagster.StringSource):

The Fivetran API secret to use for this resource.

disable_schedule_on_trigger (dagster.BoolSource, optional):

Specifies if you would like any connector that is sync’d using this resource to be automatically taken off its Fivetran schedule.

Default Value: True

request_max_retries (dagster.IntSource, optional):

The maximum number of times requests to the Fivetran API should be retried before failing.

Default Value: 3

request_retry_delay (Float, optional):

Time (in seconds) to wait between each request retry.

Default Value: 0.25

This resource allows users to programatically interface with the Fivetran REST API to launch syncs and monitor their progress. This currently implements only a subset of the functionality exposed by the API.

For a complete set of documentation on the Fivetran REST API, including expected response JSON schemae, see the Fivetran API Docs.

To configure this resource, we recommend using the configured method.

Examples:

from dagster import job
from dagster_fivetran import fivetran_resource

my_fivetran_resource = fivetran_resource.configured(
    {
        "api_key": {"env": "FIVETRAN_API_KEY"},
        "api_secret": {"env": "FIVETRAN_API_SECRET"},
    }
)

@job(resource_defs={"fivetran":my_fivetran_resource})
def my_fivetran_job():
    ...

class dagster_fivetran.FivetranResource(*, api_key, api_secret, disable_schedule_on_trigger=True, request_max_retries=3, request_retry_delay=0.25)[source]¶

This class exposes methods on top of the Fivetran REST API.

Assets¶

dagster_fivetran.load_assets_from_fivetran_instance(fivetran, key_prefix=None, connector_to_group_fn=<function _clean_name>, io_manager_key=None, connector_to_io_manager_key_fn=None, connector_filter=None, connector_to_asset_key_fn=None, poll_interval=10, poll_timeout=None)[source]¶

Loads Fivetran connector assets from a configured FivetranResource instance. This fetches information about defined connectors at initialization time, and will error on workspace load if the Fivetran instance is not reachable.

Parameters:

• fivetran (ResourceDefinition) – A FivetranResource configured with the appropriate connection details.

• key_prefix (Optional[CoercibleToAssetKeyPrefix]) – A prefix for the asset keys created.

• connector_to_group_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an asset group name for a given Fivetran connector name. If None, no groups will be created. Defaults to a basic sanitization function.

• io_manager_key (Optional[str]) – The IO manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connector_to_io_manager_key_fn.

• connector_to_io_manager_key_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an IO manager key for a given Fivetran connector name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.

• connector_filter (Optional[Callable[[FivetranConnectorMetadata], bool]]) – Optional function which takes in connector metadata and returns False if the connector should be excluded from the output assets.

• poll_interval (float) – The time (in seconds) that will be waited between successive polls.

• poll_timeout (Optional[float]) – The maximum time that will waited before this operation is timed out. By default, this will never time out.

Examples:

Loading all Fivetran connectors as assets:

from dagster_fivetran import fivetran_resource, load_assets_from_fivetran_instance

fivetran_instance = fivetran_resource.configured(
    {
        "api_key": "some_key",
        "api_secret": "some_secret",
    }
)
fivetran_assets = load_assets_from_fivetran_instance(fivetran_instance)

Filtering the set of loaded connectors:

from dagster_fivetran import fivetran_resource, load_assets_from_fivetran_instance

fivetran_instance = fivetran_resource.configured(
    {
        "api_key": "some_key",
        "api_secret": "some_secret",
    }
)
fivetran_assets = load_assets_from_fivetran_instance(
    fivetran_instance,
    connector_filter=lambda meta: "snowflake" in meta.name,
)

dagster_fivetran.build_fivetran_assets(connector_id, destination_tables, poll_interval=10, poll_timeout=None, io_manager_key=None, asset_key_prefix=None, metadata_by_table_name=None, group_name=None, infer_missing_tables=False, op_tags=None)[source]¶

Build a set of assets for a given Fivetran connector.

Returns an AssetsDefinition which connects the specified asset_keys to the computation that will update them. Internally, executes a Fivetran sync for a given connector_id, and polls until that sync completes, raising an error if it is unsuccessful. Requires the use of the fivetran_resource, which allows it to communicate with the Fivetran API.

Parameters:

• connector_id (str) – The Fivetran Connector ID that this op will sync. You can retrieve this value from the “Setup” tab of a given connector in the Fivetran UI.

• destination_tables (List[str]) – schema_name.table_name for each table that you want to be represented in the Dagster asset graph for this connection.

• poll_interval (float) – The time (in seconds) that will be waited between successive polls.

• poll_timeout (Optional[float]) – The maximum time that will waited before this operation is timed out. By default, this will never time out.

• io_manager_key (Optional[str]) – The io_manager to be used to handle each of these assets.

• asset_key_prefix (Optional[List[str]]) – A prefix for the asset keys inside this asset. If left blank, assets will have a key of AssetKey([schema_name, table_name]).

• metadata_by_table_name (Optional[Mapping[str, MetadataUserInput]]) – A mapping from destination table name to user-supplied metadata that should be associated with the asset for that table.

• group_name (Optional[str]) – A string name used to organize multiple assets into groups. This group name will be applied to all assets produced by this multi_asset.

• infer_missing_tables (bool) – If True, will create asset materializations for tables specified in destination_tables even if they are not present in the Fivetran sync output. This is useful in cases where Fivetran does not sync any data for a table and therefore does not include it in the sync output API response.

• op_tags (Optional[Dict[str, Any]]) – A dictionary of tags for the op that computes the asset. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that json.loads(json.dumps(value)) == value.

Examples:

Basic example:

from dagster import AssetKey, repository, with_resources

from dagster_fivetran import fivetran_resource
from dagster_fivetran.assets import build_fivetran_assets

my_fivetran_resource = fivetran_resource.configured(
    {
        "api_key": {"env": "FIVETRAN_API_KEY"},
        "api_secret": {"env": "FIVETRAN_API_SECRET"},
    }
)

Attaching metadata:

fivetran_assets = build_fivetran_assets(
    connector_id="foobar",
    table_names=["schema1.table1", "schema2.table2"],
    metadata_by_table_name={
        "schema1.table1": {
            "description": "This is a table that contains foo and bar",
        },
        "schema2.table2": {
            "description": "This is a table that contains baz and quux",
        },
    },
)