Awesome local LLM
Section: Inference engines · Google's on-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization
Entry
Appears in 4 awesome lists
Google's production-ready on-device ML and GenAI deployment framework. Supports Android, iOS, Web, Desktop, and IoT targets with GPU/NPU acceleration via a unified edge-first runtime. Apache 2.0 licensed.
Section: Inference engines · Google's on-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization
Section: 3. Inference Engines & Serving · Google's production-ready on-device ML and GenAI deployment framework. Supports Android, iOS, Web, Desktop, and IoT targets with GPU/NPU acceleration via a unified edge-first runtime. Apache 2.0 licensed.
Section: Deployment and Serving · LiteRT (formerly TensorFlow Lite) is Google's high-performance runtime for on-device AI inference, enabling deployment of machine learning models on mobile, embedded, and edge devices.
Section: Other · LiteRT, successor to TensorFlow Lite. is Google's On-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization
Unified proxy and SDK that routes to 100+ LLM providers behind a single OpenAI-compatible interface, with a Router handling retry/fallback across deployments, per-project cost and rate-limit tracking, and OTEL callback integrations. The right infrastructure layer when your harness needs provider…
robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video,…
Ollama is a tool for running large language models locally, offering easy setup for macOS, Windows, Linux, and Docker, along with a library of models and quickstart guides for customization and integration github | github profile
Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in inference engine for RAG, making it a powerful AI deployment solution.
Pure C/C++ inference engine with GGUF format support. The gold standard for CPU/GPU/Apple Silicon on-device running. Includes llama-server for OpenAI-compatible API. Now at 100K+ stars.
State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.
🟢🟠 — Apache-licensed AI and MCP gateway with multi-provider routing, virtual-key access controls, budgets, rate limits, MCP aggregation, OAuth, automatic fallbacks, and load balancing. (Maxim) — note: model-provider credentials and proxied request/response data are sensitive; restrict and…;…