Skip to content

Entry

Apache Airflow

Appears in 9 awesome lists

is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…

Open airflow.apache.org

Found in these lists

Awesome Apache Airflow

Section: Vital links · (also the official website)

FreshScore 81

Awesome High Performance Computing

Section: Software · A platform to programmatically author, schedule, and monitor workflows.

FreshScore 78

Awesome LLMOps

Section: Workflow · A platform to programmatically author, schedule and monitor workflows.

FreshScore 91

Awesome MLOps

Section: Data Processing · Platform to programmatically author, schedule, and monitor workflows.

FreshScore 86

Awesome Scalability

Section: Scalability

SlowScore 55

Awesome-Selfhosted

Section: Automation · Platform to programmatically author, schedule, and monitor workflows. (Source Code) Apache-2.0 Python/Docker

FreshScore 98

Awesome Software Engineering for Machine Learning

Section: Tooling · Programmatically author, schedule and monitor workflows.

StaleScore 49

Table of Contents

Section: ML frameworks & applications · is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…

SlowScore 60

Table of Contents

Section: ML Frameworks, Libraries, and Tools · is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…

SlowScore 40

Docker

is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly working in collaboration with cloud, Linux, and Windows vendors, including Microsoft.

In 20 listsDetails

Prometheus

is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting.

In 15 listsDetails

Apache Airflow

"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…

In 13 listsDetails

Prefect

Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.

In 11 listsDetails

Kubernetes

is an open-source system for automating deployment, scaling, and management of containerized applications.

In 12 listsDetails

Argo Workflows

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).

In 10 listsDetails

Vaex

Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).

In 8 listsDetails

Hamilton

| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.

In 9 listsDetails