Skip to content

Entry

Apache NiFi

Appears in 5 awesome lists

Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.

Open nifi.apache.org

Found in these lists

Awesome Big Data

Section: Data Ingestion · Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.

FreshScore 90

Awesome Data Engineering

Section: Stream Processing · An easy to use, powerful, and reliable system to process and distribute data.

FreshScore 93

Awesome ETL

Section: ETL Tools (GUI) · "a rich, web-based interface for designing, controlling, and monitoring a dataflow."

SlowScore 74

Awesome Node-Based UIs

Section: Data Processing · Data routing, transformation, and system mediation logic

SlowScore 59

Awesome Pipeline

Section: Other projects · Powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

FreshScore 82

n8n

Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.

In 10 listsDetails

CocoIndex

Incremental data transformation engine for AI pipelines with built-in chunking, embedding, and source connectors that recomputes only the inputs that changed. Apache-2.0 licensed.

In 8 listsDetails

Pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG. Features 350+ connectors with always-in-sync data from SharePoint, Google Drive, S3, Kafka, PostgreSQL and more. BSL 1.1 license (becomes Apache 2.0 after 4 years).

In 7 listsDetails

Logstash

is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

In 7 listsDetails

Apache Flink

A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.

In 6 listsDetails

Kafka

🛠 - 🐙 - Distributed, fault tolerant, high throughput pub-sub messaging system.

In 6 listsDetails

Apache Beam

Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.

In 6 listsDetails

ingestr

CLI tool to copy data from any source to any destination with a single command. Load data into your warehouse before dbt transforms it. Supports 50+ sources including PostgreSQL, MongoDB, Salesforce, Shopify.

In 6 listsDetails