Skip to content

Entry

Dagster

Appears in 12 awesome lists

Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.

Open github.comdagster-io/dagster

Found in these lists

Awesome Big Data

Section: Scheduling · a data orchestrator for machine learning, analytics, and ETL.

FreshScore 90

Awesome Data Analysis

Section: Tools · A data orchestrator for machine learning, analytics, and ETL.

FreshScore 79

Awesome Data Engineering

Section: Workflow · An open-source Python library for building data applications.

FreshScore 93

Awesome Integration

Section: Workflow Engine · Data orchestrator with a declarative, asset-based programming model for building and observing data pipelines.

FreshScore 84

Awesome MLOps

Section: Data Processing · A data orchestrator for machine learning, analytics, and ETL.

FreshScore 86

Awesome Open Source AI

Section: 8. MLOps / LLMOps & Production · Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.

FreshScore 87

Awesome Opensource Data Engineering

Section: General · ] - A data orchestrator for machine learning, analytics, and ETL.

StaleScore 36

Awesome Pipeline

Section: Pipeline frameworks & libraries · Python-based API for defining DAGs that interfaces with popular workflow managers for building data applications.

FreshScore 82

Awesome Production Machine Learning

Section: Data Pipeline · A data orchestrator for machine learning, analytics, and ETL.

FreshScore 98

awesome-python

Section: Data Science and Analytics · An orchestration platform for the development, production, and observation of data assets.

FreshScore 79

Best Of Python

Section: Dagster (32 · 16K) - An orchestration platform for the development, production, and.. Apache-2 · (👨‍💻 710 · 🔀 2.4K · 📦 5K):

FreshScore 94

Awesome Python

Section: Job Schedulers · An orchestration platform for the development, production, and observation of data assets.

FreshScore 100

Luigi

Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.

In 15 listsDetails

Apache Airflow

"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…

In 13 listsDetails

Prefect

Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.

In 11 listsDetails

Dex

Gemini CLI extension with three Agent Skills for analytics engineering: explore a warehouse or DuckDB file read-only (profiles, PII flags, verified joins), author dbt models and semantic layers as reviewable diffs, and detect schema drift. Install with gemini extensions install…

In 11 listsDetails

Apache Spark

Unified analytics engine for large-scale data processing. In-memory cluster computing with high-level APIs in Python, Scala, Java, and R. Powers MLlib for distributed machine learning and Structured Streaming for real-time data. Apache 2.0 licensed.

In 8 listsDetails

Hamilton

| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.

In 9 listsDetails

Kedro

Toolbox for production-ready data science. Uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular. Apache 2.0 licensed.

In 7 listsDetails

Apache Flink

Stream processing framework with powerful batch and streaming capabilities. High-throughput, low-latency runtime with exactly-once processing guarantees. Ideal for real-time AI inference pipelines and event-driven ML applications. Apache 2.0 licensed.

In 7 listsDetails