Awesome Apache Airflow
Section: Vital links · (latest stable release 1.10.12)
FreshScore 82
Entry
Apache Airflow
Appears in 13 awesome lists
"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
Open github.comapache/airflow
Found in these lists
Awesome Data Analysis
Section: Tools · A platform to programmatically author, schedule, and monitor workflows.
FreshScore 80
Awesome Data Engineering
Section: Workflow · A system to programmatically author, schedule, and monitor data pipelines.
FreshScore 87
AWESOME DATA SCIENCE
Section: Miscellaneous Tools · Platform to programmatically author, schedule, and monitor workflows
FreshScore 92
Awesome ETL
Section: Workflow Management/Engines · "Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
ActiveScore 72
Awesome Flask
Section: Open Source Projects · Platform to author, schedule, and monitor workflows.
FreshScore 86
Awesome Integration
Section: Workflow Engine · Platform for programmatically creating, scheduling, and monitoring workflows, ideal for managing complex data pipelines.
FreshScore 82
Awesome Open Source AI
Section: 1. Core Frameworks & Libraries · Platform to programmatically author, schedule, and monitor workflows. Industry-standard orchestration for data pipelines and ML workflows with 500+ integrations. Apache 2.0 licensed.
FreshScore 89
Awesome Production Machine Learning
Section: Data Pipeline · Data Pipeline framework built in Python, including scheduler, DAG definition and a UI for visualisation.
FreshScore 92
Awesome Python
Section: Job Schedulers · Airflow is a platform to programmatically author, schedule and monitor workflows.
FreshScore 94
Best Of Python
Section: Airflow (35 · 48K) - Platform to programmatically author, schedule, and monitor workflows. Apache-2 · (👨💻 4.7K · 🔀 18K · 📦 20K):
FreshScore 88
Awesome AI Agents: Tools, Resources, and Projects
Section: Workflows · Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
SlowScore 68
awesome-python
Section: Data Science and Analytics · Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
FreshScore 81
Often listed next to it
Luigi
Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.
In 15 listsDetails
Dagster
Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.
In 12 listsDetails
Prefect
Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.
In 11 listsDetails
Hamilton
| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.
In 10 listsDetails
Apache Airflow
is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…
In 9 listsDetails
Apache Spark
Unified analytics engine for large-scale data processing. In-memory cluster computing with high-level APIs in Python, Scala, Java, and R. Powers MLlib for distributed machine learning and Structured Streaming for real-time data. Apache 2.0 licensed.
In 8 listsDetails
Kestra
Event-driven orchestration and scheduling platform for mission-critical workflows. Infrastructure-as-Code approach with declarative YAML, Git version control integration, and hundreds of plugins for data pipelines and ML workflows. Apache 2.0 licensed.
In 8 listsDetails
Apache Flink
Stream processing framework with powerful batch and streaming capabilities. High-throughput, low-latency runtime with exactly-once processing guarantees. Ideal for real-time AI inference pipelines and event-driven ML applications. Apache 2.0 licensed.
In 7 listsDetails