Awesome Big Data
Section: Distributed Programming · A fast and simple framework for building and running distributed applications.
Entry
Appears in 13 awesome lists
A fast and simple framework for building and running distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library. ray.io
Section: Distributed Programming · A fast and simple framework for building and running distributed applications.
Section: Miscellaneous Tools · Scalable hyperparameter tuning library
Section: Python · RLlib is an industry level, highly scalable RL library for tf and torch, based on Ray. It's used by companies like Amazon and Microsoft to solve real-world decision making problems at scale.
Section: Optimization Tools · Fast and simple framework for building and running distributed applications.
Section: 7. Training & Fine-tuning Ecosystem · Scalable distributed training.
Section: Pipeline frameworks & libraries · Flexible, high-performance distributed Python execution framework.
Section: Computation and Communication Optimisation · Ray is a flexible, high-performance distributed execution framework for machine learning.
Section: Concurrency and Parallelism · Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Section: Other libraries: · A fast and simple framework for building and running distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library. ray.io
Section: Machine Learning · A fast and simple framework for building and running distributed applications.
Section: Graph Computation · An open source framework that provides a simple, universal API for building distributed applications.
Section: Distributed Computing · A unified framework for scaling AI and Python applications.
Section: Computation · | Python, C++ | - An open source framework that provides a simple, universal API for building distributed applications.
How to use the Hexagon Delegate to speed up model inference on mobile and edge devices. Also see blog post Accelerating TensorFlow Lite on Qualcomm Hexagon DSPs.
(label: good first issue) PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.
Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…
(formerly known as pytorch-transformers and pytorch-pretrained-bert) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in…
(from Hpcaitech) - A Unified Deep Learning System for Large-Scale Parallel Training (1D, 2D, 2.5D, 3D and sequence parallelism, and ZeRO protocol).
"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search,…