Skip to content

Entry

Delta Lake

Appears in 6 awesome lists

Open-source storage framework enabling Lakehouse architecture with ACID transactions, scalable metadata handling, and unified batch/streaming processing. Apache 2.0 licensed.

Open github.comdelta-io/delta

Found in these lists

Awesome Data Analysis

Section: Tools · A storage layer that brings ACID transactions to Apache Spark and big data workloads.

FreshScore 79

Awesome LLMOps

Section: Data Management · Storage layer that brings scalable, ACID transactions to Apache Spark and other engines.

FreshScore 91

Awesome MLOps

Section: Data Management · Storage layer that brings scalable, ACID transactions to Apache Spark and other engines.

FreshScore 86

Awesome Open Source AI

Section: 1. Core Frameworks & Libraries · Open-source storage framework enabling Lakehouse architecture with ACID transactions, scalable metadata handling, and unified batch/streaming processing. Apache 2.0 licensed.

FreshScore 87

Awesome Production Machine Learning

Section: Data Storage Optimisation · Delta Lake is a storage layer that brings scalable, ACID transactions to Apache Spark and other big-data engines.

FreshScore 98

Awesome Spark

Section: Storage · Storage layer with ACID transactions.

SlowScore 61

Luigi

Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.

In 15 listsDetails

Milvus

Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.

In 14 listsDetails

Apache Airflow

"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…

In 13 listsDetails

Dagster

Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.

In 12 listsDetails

Prefect

Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.

In 11 listsDetails

Dolt

Dolt is Git for Data! Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a git repository.

In 10 listsDetails

lakeFS

Data version control for your data lake that transforms object storage into Git-like repositories. Enables atomic, versioned data lake operations with branching, committing, and merging for data pipelines. Apache 2.0 licensed.

In 10 listsDetails

Dex

Gemini CLI extension with three Agent Skills for analytics engineering: explore a warehouse or DuckDB file read-only (profiles, PII flags, verified joins), author dbt models and semantic layers as reviewable diffs, and detect schema drift. Install with gemini extensions install…

In 11 listsDetails