Awesome LLM Resources
Section: 小多模态模型 Small Vision Language Model
FreshScore 87
Entry
Nano-vLLM
Appears in 5 awesome lists
Minimalist vLLM implementation in ~1,200 lines of Python. Educational yet performant with prefix caching, tensor parallelism, and CUDA graph acceleration. Comparable inference speeds to full vLLM. MIT licensed.
Open github.comgeeeekexplorer/nano-vllm
Found in these lists
Awesome local LLM
Section: Inference engines · a lightweight vLLM implementation built from scratch
FreshScore 87
Awesome Open Source AI
Section: 3. Inference Engines & Serving · Minimalist vLLM implementation in ~1,200 lines of Python. Educational yet performant with prefix caching, tensor parallelism, and CUDA graph acceleration. Comparable inference speeds to full vLLM. MIT licensed.
FreshScore 89
Awesome Production Machine Learning
Section: Deployment and Serving · nano-vllm is a lightweight vLLM implementation built from scratch, providing fast offline inference with optimization techniques such as prefix caching, tensor parallelism, and CUDA graph.
FreshScore 92
awesome-python
Section: Other · Nano vLLM
FreshScore 81
Often listed next to it
Ollama
Ollama is a tool for running large language models locally, offering easy setup for macOS, Windows, Linux, and Docker, along with a library of models and quickstart guides for customization and integration github | github profile
In 12 listsDetails
vLLM
State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.
In 11 listsDetails
llama.cpp
Pure C/C++ inference engine with GGUF format support. The gold standard for CPU/GPU/Apple Silicon on-device running. Includes llama-server for OpenAI-compatible API. Now at 100K+ stars.
In 10 listsDetails
OpenLLM
Production-grade platform for running any open-source LLMs as OpenAI-compatible API endpoints. Supports 50+ models with built-in streaming, batching, and auto-acceleration. Apache 2.0 licensed.
In 10 listsDetails
SGLang
(MPL-2.0) allows specifying JSON schemas using regular expressions or Pydantic models for constrained decoding. Its high-performance runtime accelerates JSON decoding.
In 9 listsDetails
Bifrost
🟢🟠 — Apache-licensed AI and MCP gateway with multi-provider routing, virtual-key access controls, budgets, rate limits, MCP aggregation, OAuth, automatic fallbacks, and load balancing. (Maxim) — note: model-provider credentials and proxied request/response data are sensitive; restrict and…;…
In 8 listsDetails
shimmy
Python-free Rust inference server with OpenAI API compatibility. Supports GGUF and SafeTensors formats with hot model swap, auto-discovery, and single binary deployment for zero-dependency inference. Apache 2.0 licensed.
In 7 listsDetails
TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference…
In 7 listsDetails