Awesome Data Engineering
Section: Stream Processing · A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
Entry
Appears in 6 awesome lists
A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
Section: Stream Processing · A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
Section: Miscellaneous Tools · A platform for efficient, distributed, general-purpose data processing.
Section: Big Data (Hadoop Stack) · "a framework and distributed processing engine for stateful computations over unbounded and bounded data streams."
Section: Realtime Data Processing · Apache Flink is a platform for efficient, distributed, general-purpose data processing. It supports exactly once stream processing.
Section: Java · Open source platform for distributed stream and batch data processing.
Section: 缓存、消息队列与流处理 (Caching, MQ & Streaming)
Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…
A fast and simple framework for building and running distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library. ray.io
"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
industrial-grade ASR toolkit; 170× realtime on GPU, 50+ languages, built-in VAD, punctuation, speaker diarization, and emotion detection. Includes non-autoregressive SenseVoice and LLM-based Fun-ASR-Nano models.
Build and share delightful machine learning apps, all in Python. The de facto standard for creating interactive ML demos with automatic UI generation from function signatures. Powers thousands of Hugging Face Spaces.
The fastest way to build and share data apps. Transform Python scripts into beautiful web applications with minimal code. Widely used for ML model demos, data visualization, and internal tools.