Awesome LLMOps
Section: Optimizations · KV cache layer for vLLM and SGLang that stores and reuses KV caches across GPU memory, CPU memory, local disk and remote storage, cutting time to first token for long contexts and multi-turn chats.
Entry
Appears in 5 awesome lists
Supercharge LLM inference with the fastest KV Cache layer. 3-10x delay savings and GPU cycle reduction for multi-round QA and RAG. Integrates seamlessly with vLLM for distributed, high-throughput deployments. Apache 2.0 licensed.
Section: Optimizations · KV cache layer for vLLM and SGLang that stores and reuses KV caches across GPU memory, CPU memory, local disk and remote storage, cutting time to first token for long contexts and multi-turn chats.
Section: Memory Management · supercharge your LLM with the fastest KV Cache Layer
Section: 3. Inference Engines & Serving · Supercharge LLM inference with the fastest KV Cache layer. 3-10x delay savings and GPU cycle reduction for multi-round QA and RAG. Integrates seamlessly with vLLM for distributed, high-throughput deployments. Apache 2.0 licensed.
Section: Deployment and Serving · LMCache is a high-performance KV cache layer that accelerates LLM inference.
Section: Other · LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
The most-deployed memory layer for AI agents — drop-in add/search/update/delete fact storage with LLM-extracted long-term / short-term / semantic memory: automatic fact consolidation (dedup, contradiction resolution, decay) so agents keep "context that persists" without context bloat;…
Ollama is a tool for running large language models locally, offering easy setup for macOS, Windows, Linux, and Docker, along with a library of models and quickstart guides for customization and integration github | github profile
Pure C/C++ inference engine with GGUF format support. The gold standard for CPU/GPU/Apple Silicon on-device running. Includes llama-server for OpenAI-compatible API. Now at 100K+ stars.
State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.
🟢🟠 — Apache-licensed AI and MCP gateway with multi-provider routing, virtual-key access controls, budgets, rate limits, MCP aggregation, OAuth, automatic fallbacks, and load balancing. (Maxim) — note: model-provider credentials and proxied request/response data are sensitive; restrict and…;…
Production-grade platform for running any open-source LLMs as OpenAI-compatible API endpoints. Supports 50+ models with built-in streaming, batching, and auto-acceleration. Apache 2.0 licensed.
(MPL-2.0) allows specifying JSON schemas using regular expressions or Pydantic models for constrained decoding. Its high-performance runtime accelerates JSON decoding.
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference…