Skip to content
36

Awesome Opensource Data Engineering

An Awesome List of Open-Source Data Engineering Projects

3.3k stars569 forks84 entriesLast push Oct 4, 2024 (2 years ago)License Other

This page lists names, links and short descriptions. The original list on GitHub is the source and belongs to its authors.

General

This

List] aims at providing an overview of https://opensource.org/licenses[open-source] projects related to data engineering. This is a community effort: please https://github.com/gunnarmorling/awesome-opensource-data-engineering/blob/master/CONTRIBUTING.md[contribute] and send your pull requests for…

https://spark.apache.org/[Apache

Spark] - A unified analytics engine for large-scale data processing. Includes APIs in Scala, Java, Python (known as PySpark), and R (SparkR).

Link may be dead

https://beam.apache.org/[Apache

Beam] - An open-source implementation of Google DataFlow. Provides capabilites of batch and streaming data processing jobs that run on any execution engine, including Spark, Flink, or its own DirectRunner. Supports multiple APIs in Java, Python, and Go.

Link may be dead

https://flink.apache.org/[Apache

Flink] - Stateful computations over data streams.

Link may be dead

https://trino.io/[Trino

(formerly known as PrestoSQL)] - Distributed SQL Query Engine for Big Data.

Link may be dead

https://superset.incubator.apache.org/[Apache

Superset] - A modern, enterprise-ready business intelligence web application.

Link may be dead

https://gethue.com/[HUE

] - The Hadoop User Interface. Similar to Superset, but interfaces between RDBMS, Hive, Impala, HBase, Spark, HDFS & S3, Oozie, Pig, YARN Job Explorer, and more. Offers an extensible Django environment for custom app integration.

Link may be dead

https://www.metabase.com/[Metabase

] - An easy way for everyone in your company to ask questions and learn from data.

Link may be dead

https://redash.io/[Redash

] - All the tools to unlock your data.

Link may be dead

https://delta.io/[Delta

Lake] - Open-source storage framework that enables building a lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python.

Link may be dead

https://hudi.apache.org/[Apache

Hudi] - Transactional data lake platform that brings database and data warehouse capabilities to the data lake. Hudi reimagines slow old-school batch data processing with a powerful new incremental processing framework for low latency minute-level analytics.

Link may be dead

https://iceberg.apache.org/[Apache

Iceberg] - High-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.

Link may be dead

https://debezium.io/[Debezium

] - Change data capture for MySQL, Postgres, MongoDB, SQL Server and others.

Link may be dead

https://github.com/zendesk/maxwell[Maxwell

] - Maxwell's daemon, a MySQL-to-JSON Kafka producer.

https://calcite.apache.org/[Apache

Calcite] - SQL parser, building blocks for datastores.

Link may be dead

http://cassandra.apache.org/[Apache

Cassandra] - Open Source distributed wide column store, NoSQL database.

Link may be dead

https://druid.apache.org/[Apache

Druid] - A high performance real-time analytics database.

Link may be dead

https://hbase.apache.org/[Apache

HBase] - Open Source non-relational distributed database.

Link may be dead

https://pinot.apache.org/[Apache

Pinot] - A realtime distributed OLAP datastore.

Link may be dead

https://clickhouse.tech/[ClickHouse

] - Open Source distributed column-oriented DBMS.

Link may be dead

https://www.influxdata.com/[InfluxDB

] - Purpose-Built Open Source Time Series Database.

https://min.io/[MinIO

] - MinIO is a high performance, distributed object storage system and AWS S3 compatible.

Link may be dead

https://www.postgresql.org/[Postgres

] - The World's Most Advanced Open Source Relational Database.

Link may be dead

https://questdb.io/[QuestDB

] - Open Source Time Series Database with a focus on performance and simplicity.

Link may be dead

https://github.com/lyft/amundsen[Amundsen

] - metadata catalogue.

https://github.com/linkedin/datahub[DataHub

] - A Generalized Metadata Search & Discovery Tool.

https://github.com/Netflix/metacat[Metacat

] - Unified metadata exploration API service.

https://github.com/elementary-data/elementary-lineage[Elementary

] - Data reliability solution, starting with plug-and-play data lineage and datasets operational status.

https://github.com/monosidev/monosi[Monosi

] - Data observability & monitoring platform.

https://github.com/open-metadata/OpenMetadata[OpenMetadata

] - Generalized metadata, search, and lineage tool.

https://drill.apache.org/[Apache

Drill] - Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.

Link may be dead

https://github.com/dremio/dremio-oss[Dremio

] - A data lake engine. Provides an Apache Arrow-based query and acceleration engine together with the ability to create an IT-governed self-service layer for data scientists and analysts.

http://teiid.io/[Teiid

] - A relational abstraction of different information sources.

Link may be dead

https://prestodb.io/[Presto

] - Distributed SQL Query Engine for Big Data.

Link may be dead

https://github.com/Alluxio/alluxio[Alluxio

] - Scalable, multi-tiered distributed caching for HDFS, S3, Ceph, NFS, and related filestores. Provides integrations for SQL queries into a Catalog from Spark, Hive, and Presto.

https://www.getdbt.com/[dbt

] - Empowering data analysts and engineers to apply methodologies akin to those used by software engineers for constructing applications, dbt ensures data transformation processes align with established practices.

Link may be dead

https://avro.apache.org/[Apache

Avro] - A data serialization system.

Link may be dead

https://parquet.apache.org/[Apache

Parquet] - A columnar storage format.

Link may be dead

https://orc.apache.org/[Apache

ORC] - Another columnar storage format.

Link may be dead

https://thrift.apache.org/[Apache

Thrift] - Data type and service interface definitions and code generator.

Link may be dead

https://arrow.apache.org/[Apache

Arrow] - A cross-language development platform for in-memory data. It specifies a standardized, language-independent, columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy IPC…

Link may be dead

https://capnproto.org/[Cap’n

Proto] - A data interchange format and capability-based RPC system.

Link may be dead

https://google.github.io/flatbuffers/[FlatBuffers

] - An efficient cross platform serialization library for C++, C#, C, Go, Java, JavaScript, Lobster, Lua, TypeScript, PHP, Python, and Rust.

Link may be dead

https://msgpack.org/index.html[MessagePack

] - An efficient binary serialization format. It lets you exchange data among multiple languages like JSON.

Link may be dead

https://developers.google.com/protocol-buffers[Protocol

Buffers] - Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data.

Link may be dead

https://camel.apache.org/[Apache

Camel] - Easily integrate various systems consuming or producing data.

Link may be dead

https://kafka.apache.org/documentation/#connect[Kafka

Connect] - Reusable framework to handle data int-and-out of Apache Kafka.

In 4 lists

https://www.elastic.co/logstash[Logstash

] - Open Source server-side data processing pipeline.

Link may be dead

https://github.com/influxdata/telegraf[Telegraf

] - a plugin-driven server agent writen in Go (deployed as a single binary with no external dependencies) for collecting and sending metrics and events from databases, systems, and IoT sensors. Offers hundreds of existing plugins.

https://activemq.apache.org/[Apache

ActiveMQ] - Flexible & Powerful Multi-Protocol Messaging.

Link may be dead

https://kafka.apache.org/[Apache

Kafka] - A distributed commit log with messaging capabilities.

Link may be dead

https://pulsar.apache.org/[Apache

Pulsar] - A distributed pub-sub messaging system.

Link may be dead

http://github.com/bsideup/liiklus[Liiklus

] - An event gateway that provides reactive gRPC/RSocket access to Kafka-like systems.

https://nakadi.io/[Nakadi

] - A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues].

Link may be dead

https://nats.io/[NATS

] - A simple, secure and high performance messaging system.

Link may be dead

https://www.rabbitmq.com/[RabbitMQ

] - A message broker.

Link may be dead

https://github.com/wepay/waltz[Waltz

] - A quorum-based distributed write-ahead log for replicating transactions.

https://zeromq.org/[ZeroMQ

] - An open-source universal, high-performance messaging library.

Link may be dead

https://cloudevents.io/[CloudEvents

] - A specification for describing event data in a common way.

Link may be dead

https://kafka.apache.org/documentation/streams/[Apache

Kafka Streams] - A client library for building applications and microservices, where the input and output data are stored in Kafka.

Link may be dead

http://samza.apache.org/[Apache

Samza] - A distributed stream processing framework.

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html[Apache

Spark Structured Streaming] - A scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

Link may be dead

http://storm.apache.org/[Apache

Storm] - A distributed realtime computation system.

Link may be dead

https://greatexpectations.io/[Great

expectations] - Helps data teams eliminate pipeline debt, through data testing.

Link may be dead

https://github.com/DataKitchen/data-observability-installer/[DataKitchen

Data Observability] - A full featured data quality profiling and data testing tool: it automatically generates tests for you.

In 2 lists

https://prometheus.io/[prometheus

] - An open-source systems monitoring and alerting toolkit.

Link may be dead

https://grafana.com/[grafana

] - An open-source analytics and monitoring platform.

Link may be dead

https://github.com/treeverse/lakeFS/[lakeFS

] - Repeatable, atomic and versioned data lake on top of object storage.

In 10 listsDetails

https://github.com/meirwah/awesome-workflow-engines[Awesome

Workflow Engines] - A curated list of awesome open source workflow engines.

https://airflow.apache.org/[Apache

Airflow] - A platform created by community to programmatically author, schedule and monitor workflows.

https://nifi.apache.org/[Apache

NiFi] - Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic

Link may be dead

https://github.com/knime/[KNIME

] - KNIME Analytics Platform offers a WYSIWYG Editor for Spark-based workflows, with over 2000+ integrations. Offers visualization and flow analytics in-place. KNIME Server is a commercially licensed component that adds additional features.

https://github.com/PrefectHQ/prefect/[Prefect

] - A workflow management system designed for modern infrastructure.

In 11 listsDetails

https://github.com/dagster-io/dagster/[Dagster

] - A data orchestrator for machine learning, analytics, and ETL.

In 12 listsDetails

https://github.com/kestra-io/kestra[Kestra

] - Open source data orchestration and scheduling platform with declarative syntax.

https://github.com/mage-ai/mage-ai[Mage

] - Open source data orchestration and scheduling platform with a rich interactive UI for workflows.

https://www.dataengineeringpodcast.com/[Data

Engineering Podcast]

Link may be dead

https://softwareengineeringdaily.com/[Software

Engineering Daily]

https://datastackshow.com/[Data

Stack Show]

Link may be dead

https://dataengweekly.substack.com/[Data

Eng Weekly]

Link may be dead

https://nosql-database.org/[NOSQL

Database Management Systems] - List of NoSQL database management systems.

https://db-engines.com/en/[DB-Engines

] - Knowledge base of relational and NoSQL database management systems.

Link may be dead

https://www.goodreads.com/list/show/146550.Data_Engineering_Group[Books

] and https://www.goodreads.com/group/show/1073364-data-engineering[Book club] - Goodreads list and group about Data Engineering books

https://www.kdnuggets.com/25-free-books-to-master-sql-python-data-science-machine-learning-and-natural-language-process…

Free Data Books] - Collection of 25 free e-books related to SQL, Python, Data Science, Machine Learning, and Natural Language Processing.

Link may be dead
See category
99

Awesome DeepSeek Harness (DSH) Plugin

awesome-dsh-plugin/awesome-dsh-plugin

A curated list of plugins for DeepSeek Harness (dsh) · DeepSeek Harness 插件精选列表

Fresh★ 25k1416 entriesPushed today
98

Awesome OpenClaw Skills

VoltAgent/awesome-openclaw-skills

The awesome collection of OpenClaw skills. 5,400+ skills filtered and categorized from the official OpenClaw Skills Registry.🦞

Fresh★ 54k843 entriesPushed 16 days ago
96

Awesome Prompts

ai-boost/awesome-prompts

Curated prompts, frameworks and papers for LLMs and agents, with an engineering bias. Prompt engineering, context engineering, agent harnesses, red teaming.

Fresh★ 9.3k365 entriesPushed today
96

Awesome Open Source Alternatives

piotrkulpinski/open-source-alternatives

Curated list of open source alternatives to proprietary software.

Fresh★ 6.9k860 entriesPushed today
95

Awesome README

matiassingers/awesome-readme

A curated list of awesome READMEs

Fresh★ 22k144 entriesPushed 5 days ago
94

Awesome Persona Distill Skills

xixu-me/awesome-persona-distill-skills

Curated list of Agent Skills centered on people, relationships, commemorative scenes, and methodological perspectives

Fresh★ 4.7k72 entriesPushed today