Dagster
Open-source orchestration platform with software-defined assets approach for data and ML pipelines.
Dagster orchestrates pipelines as software-defined assets with declarative lineage and integrated data quality – the most modern alternative to Airflow.
Explanation
Dagster models pipelines as "assets" (e.g., tables, ML models) instead of tasks. This enables declarative lineage, automatic materialization, and integrated data quality checks.
Marketing Relevance
Dagster gains traction as an asset-centric alternative to Airflow, especially with modern data teams.
Common Pitfalls
Smaller community than Airflow. Asset paradigm requires rethinking. Fewer production experience reports.
Origin & History
Nick Schrock (formerly Facebook/GraphQL) founded Elementl and released Dagster in 2019. The software-defined assets concept was introduced in 2022. Dagster Cloud offers managed hosting. The asset-centric philosophy influences the entire orchestration landscape.
Comparisons & Differences
Dagster vs. Apache Airflow
Airflow is task-centric (what runs when); Dagster is asset-centric (what is produced).
Dagster vs. dbt
dbt transforms SQL data; Dagster orchestrates the entire pipeline lifecycle including dbt integration.