Awesome Big Data
Section: Data Ingestion · Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
Entry
Appears in 5 awesome lists
Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
Section: Data Ingestion · Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
Section: Stream Processing · An easy to use, powerful, and reliable system to process and distribute data.
Section: ETL Tools (GUI) · "a rich, web-based interface for designing, controlling, and monitoring a dataflow."
Section: Data Processing · Data routing, transformation, and system mediation logic
Section: Other projects · Powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.
Incremental data transformation engine for AI pipelines with built-in chunking, embedding, and source connectors that recomputes only the inputs that changed. Apache-2.0 licensed.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG. Features 350+ connectors with always-in-sync data from SharePoint, Google Drive, S3, Kafka, PostgreSQL and more. BSL 1.1 license (becomes Apache 2.0 after 4 years).
is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.
A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
🛠 - 🐙 - Distributed, fault tolerant, high throughput pub-sub messaging system.
Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.