Skip to content

Entry

LMCache

Appears in 5 awesome lists

Supercharge LLM inference with the fastest KV Cache layer. 3-10x delay savings and GPU cycle reduction for multi-round QA and RAG. Integrates seamlessly with vLLM for distributed, high-throughput deployments. Apache 2.0 licensed.

Open github.comlmcache/lmcache

Found in these lists

Awesome LLMOps

Section: Optimizations · KV cache layer for vLLM and SGLang that stores and reuses KV caches across GPU memory, CPU memory, local disk and remote storage, cutting time to first token for long contexts and multi-turn chats.

FreshScore 91

Awesome local LLM

Section: Memory Management · supercharge your LLM with the fastest KV Cache Layer

FreshScore 90

Awesome Open Source AI

Section: 3. Inference Engines & Serving · Supercharge LLM inference with the fastest KV Cache layer. 3-10x delay savings and GPU cycle reduction for multi-round QA and RAG. Integrates seamlessly with vLLM for distributed, high-throughput deployments. Apache 2.0 licensed.

FreshScore 87

Awesome Production Machine Learning

Section: Deployment and Serving · LMCache is a high-performance KV cache layer that accelerates LLM inference.

FreshScore 98

awesome-python

Section: Other · LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

FreshScore 79

mem0

The most-deployed memory layer for AI agents — drop-in add/search/update/delete fact storage with LLM-extracted long-term / short-term / semantic memory: automatic fact consolidation (dedup, contradiction resolution, decay) so agents keep "context that persists" without context bloat;…

In 14 listsDetails

Ollama

Ollama is a tool for running large language models locally, offering easy setup for macOS, Windows, Linux, and Docker, along with a library of models and quickstart guides for customization and integration github | github profile

In 13 listsDetails

llama.cpp

Pure C/C++ inference engine with GGUF format support. The gold standard for CPU/GPU/Apple Silicon on-device running. Includes llama-server for OpenAI-compatible API. Now at 100K+ stars.

In 12 listsDetails

vLLM

State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.

In 11 listsDetails

Bifrost

🟢🟠 — Apache-licensed AI and MCP gateway with multi-provider routing, virtual-key access controls, budgets, rate limits, MCP aggregation, OAuth, automatic fallbacks, and load balancing. (Maxim) — note: model-provider credentials and proxied request/response data are sensitive; restrict and…;…

In 11 listsDetails

OpenLLM

Production-grade platform for running any open-source LLMs as OpenAI-compatible API endpoints. Supports 50+ models with built-in streaming, batching, and auto-acceleration. Apache 2.0 licensed.

In 10 listsDetails

SGLang

(MPL-2.0) allows specifying JSON schemas using regular expressions or Pydantic models for constrained decoding. Its high-performance runtime accelerates JSON decoding.

In 9 listsDetails

TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference…

In 8 listsDetails