Skip to content

Overview

Dagster is an asset-centric workflow orchestrator.

  • Orchestrator: Tools that allow you to automatically execute steps in a specified order and track the outcomes.
  • Asset-centric: Focused on how outputs interact with other outputs. With Dagster, you declare functions in Python that you want to run and the data assets that those functions produce or update (Source).

Why use Dagster

Dagster makes it easy to:

  1. Understand how an asset is produced.
  2. Determine if an asset is up to date.
  3. Diagnose data quality issues by building data quality checks into your pipelines.
  4. Debug with live logs as the pipeline runs.
  5. View the outputs and configurations for historical runs.

Plus, Dagster is open-source and integrates easily with Azure and Docker.

Helpful resources