Skip to content

Entry

Apache Airflow

Appears in 13 awesome lists

"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…

Open github.comapache/airflow

Found in these lists

Awesome Apache Airflow

Section: Vital links · (latest stable release 1.10.12)

FreshScore 81

Awesome Data Analysis

Section: Tools · A platform to programmatically author, schedule, and monitor workflows.

FreshScore 79

Awesome Data Engineering

Section: Workflow · A system to programmatically author, schedule, and monitor data pipelines.

FreshScore 93

AWESOME DATA SCIENCE

Section: Miscellaneous Tools · Platform to programmatically author, schedule, and monitor workflows

FreshScore 93

Awesome ETL

Section: Workflow Management/Engines · "Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…

SlowScore 74

Awesome Flask

Section: Open Source Projects · Platform to author, schedule, and monitor workflows.

ActiveScore 83

Awesome Integration

Section: Workflow Engine · Platform for programmatically creating, scheduling, and monitoring workflows, ideal for managing complex data pipelines.

FreshScore 84

Awesome Open Source AI

Section: 1. Core Frameworks & Libraries · Platform to programmatically author, schedule, and monitor workflows. Industry-standard orchestration for data pipelines and ML workflows with 500+ integrations. Apache 2.0 licensed.

FreshScore 87

Awesome Production Machine Learning

Section: Data Pipeline · Data Pipeline framework built in Python, including scheduler, DAG definition and a UI for visualisation.

FreshScore 98

awesome-python

Section: Data Science and Analytics · Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

FreshScore 79

Best Of Python

Section: Airflow (33 · 48K · ) - Platform to programmatically author, schedule, and monitor.. Apache-2 · (👨‍💻 4.8K · 🔀 18K · 📦 29):

FreshScore 94

Awesome AI Agents: Tools, Resources, and Projects

Section: Workflows · Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

SlowScore 66

Awesome Python

Section: Job Schedulers · Airflow is a platform to programmatically author, schedule and monitor workflows.

FreshScore 100

Luigi

Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.

In 15 listsDetails

Dagster

Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.

In 12 listsDetails

Prefect

Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.

In 11 listsDetails

Dex

Gemini CLI extension with three Agent Skills for analytics engineering: explore a warehouse or DuckDB file read-only (profiles, PII flags, verified joins), author dbt models and semantic layers as reviewable diffs, and detect schema drift. Install with gemini extensions install…

In 11 listsDetails

Apache Airflow

is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…

In 9 listsDetails

Apache Spark

Unified analytics engine for large-scale data processing. In-memory cluster computing with high-level APIs in Python, Scala, Java, and R. Powers MLlib for distributed machine learning and Structured Streaming for real-time data. Apache 2.0 licensed.

In 8 listsDetails

Hamilton

| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.

In 9 listsDetails

Kedro

Toolbox for production-ready data science. Uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular. Apache 2.0 licensed.

In 7 listsDetails