API Reference¶

Bases: ConfigurableIOManager

An IOManager that stores directories and files on ADLS2.

Assets should return a local pathlib.Path pointing to either a file or a directory. The IOManager will upload the file or directory to ADLS2 and make it available to downstream assets.

Downstream assets receive a local pathlib.Path pointing to the downloaded file or directory. If the downstream asset's type annotation is str, the ADLS2 path is returned directly without downloading (e.g. abfss://container@account.dfs.core.windows.net/...).

Partitioned assets are fully supported. When loading multiple partitions, the downstream asset receives a Dict[str, Path] mapping partition keys to local paths, consistent with Dagster's built-in IOManager behavior.

Parameters:

Example:

.. code-block:: python

    from dagster import Definitions, asset
    from pathlib import Path

    @asset(io_manager_key="adls2_dir")
    def my_asset() -> Path:
        out = Path("/tmp/my_output")
        out.mkdir(exist_ok=True)
        (out / "results.csv").write_text("a,b,c")
        return out

    @asset(io_manager_key="adls2_dir")
    def downstream_asset(my_asset: Path) -> None:
        df = pd.read_csv(my_asset / "results.csv")

    @asset(io_manager_key="adls2_dir")
    def path_only_asset(my_asset: str) -> None:
        # receives the raw ADLS2 path without downloading
        print(my_asset)  # abfss://container@account.dfs.core.windows.net/dagster/my_asset


    defs = Definitions(
        assets=[my_asset, downstream_asset, path_only_asset],
        resources={
            "adls2_dir": ADLS2FilesystemIOManager()
        },
    )

Bases: UPathIOManager

An IOManager that stores directories and files on ADLS2.

Assets should return a local pathlib.Path pointing to either a file or a directory. The IOManager will upload the file or directory to ADLS2 and make it available to downstream assets.

Downstream assets receive a local pathlib.Path pointing to the downloaded file or directory. If the downstream asset's type annotation is str, the ADLS2 path is returned directly without downloading (e.g. abfss://container@account.dfs.core.windows.net/...).

Partitioned assets are fully supported. When loading multiple partitions, the downstream asset receives a Dict[str, Path] mapping partition keys to local paths, consistent with Dagster's built-in IOManager behavior.

Parameters:

Example:

.. code-block:: python

    from azure.storage.filedatalake import DataLakeServiceClient
    from dagster import Definitions, asset
    from pathlib import Path

    @asset(io_manager_key="adls2_dir")
    def my_asset() -> Path:
        out = Path("/tmp/my_output")
        out.mkdir(exist_ok=True)
        (out / "results.csv").write_text("a,b,c")
        return out

    @asset(io_manager_key="adls2_dir")
    def downstream_asset(my_asset: Path) -> None:
        df = pd.read_csv(my_asset / "results.csv")

    @asset(io_manager_key="adls2_dir")
    def path_only_asset(my_asset: str) -> None:
        # receives the raw ADLS2 path without downloading
        print(my_asset)  # abfss://container@account.dfs.core.windows.net/dagster/my_asset

    adls2_client = DataLakeServiceClient(
        account_url="https://<account>.dfs.core.windows.net",
        credential="<credential>",
    )

    defs = Definitions(
        assets=[my_asset, downstream_asset, path_only_asset],
        resources={
            "adls2_dir": ADLS2FilesystemIOManager(
                file_system="my-container",
                adls2_client=adls2_client,
            )
        },
    )

Represents a path in ADLS2.

This object provides authenticated access to the underlying Azure SDK clients while also exposing convenience methods for common operations.

Users can always drop down to the Azure SDK if they need functionality beyond the convenience methods.

Examples¶

Download entire asset:

local_dir = data.download()

Download only a subfolder:

parquet_dir = data.download("parquet")

Access Azure SDK directly:

directory_client = data.get_directory_client()

for item in directory_client.get_paths():
    ...

data.download()

data.download("parquet")

data.download("parquet/2026/01")

Bases: ConfigurableIOManager

Persistent IO manager using Azure Data Lake Storage Gen2 for storage.

Serializes objects via pickling. Suitable for objects storage for distributed executors, so long as each execution node has network connectivity and credentials for ADLS and the backing container.

Assigns each op output to a unique filepath containing run ID, step key, and output name. Assigns each asset to a single filesystem path, at "/". If the asset key has multiple components, the final component is used as the name of the file, and the preceding components as parent directories under the base_dir.

Subsequent materializations of an asset will overwrite previous materializations of that asset. With a base directory of "/my/base/path", an asset with key AssetKey(["one", "two", "three"]) would be stored in a file called "three" in a directory with path "/my/base/path/one/two/".

Example usage:

1. Attach this IO manager to a set of assets.

.. code-block:: python

from dagster import Definitions, asset
from cfa_dagster import ADLS2PickleIOManager

@asset
def asset1():
    # create df ...
    return df

@asset
def asset2(asset1):
    return df[:5]

Definitions(
    assets=[asset1, asset2],
    resources={
        "io_manager": ADLS2PickleIOManager(),
    },
)

1. Attach this IO manager to your job to make it available to your ops.

.. code-block:: python

from dagster import job
from cfa_dagster import ADLS2PickleIOManager

@job(
    resource_defs={
        "io_manager": ADLS2PickleIOManager(),
    },
)
def my_job():
    ...

Bases: StepHandler

Bases: StepHandler

Bases: RunLauncher, ConfigurableClass

Launches runs in an Azure Container App Job.

Bases: ConfigurableResource

Bases: Executor

Bases: QueuedRunCoordinator

Bases: RunLauncher, ConfigurableClass

Launches a run using a runtime-configurable launcher

Bases: StepHandler

Bases: StepHandler

Executes each step in a subprocess. Supports the same config as multiprocess_executor. Note: tag_concurrency_limits is not supported — it is implemented at the StepDelegatingExecutor level and cannot be enforced per StepHandler.

Bases: StepHandler

Executes steps one at a time by blocking until each subprocess completes. Unlike SubprocessStepHandler, this runs steps synchronously so the next step cannot start until this one finishes — useful for lightweight ops that should not incur container spin-up cost but also should not run concurrently with their dependents.