Skip to content

Entry

tokenizers

Appears in 6 awesome lists

Hugging Face's tokenizers for modern NLP pipelines (original implementation) with bindings for Python.

Open github.comhuggingface/tokenizers

Found in these lists

awesome-huggingface

Section: Official Libraries · Fast state-of-the-Art tokenizers optimized for research and production.

ArchivedScore 55

Awesome LLMOps

Section: Large Model Serving · 💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

FreshScore 91

Awesome Machine Learning

Section: Rust · Fast State-of-the-Art Tokenizers optimized for Research and Production

FreshScore 93

awesome-nlp

Section: Libraries · Tokenizers optimized for Research and Production.

FreshScore 91

Awesome Open Source AI

Section: 1. Core Frameworks & Libraries · Fast state-of-the-art tokenizers for training and inference.

FreshScore 87

Awesome Rust

Section: Artificial Intelligence · Hugging Face's tokenizers for modern NLP pipelines (original implementation) with bindings for Python.

FreshScore 81

transformers

(formerly known as pytorch-transformers and pytorch-pretrained-bert) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in…

In 14 listsDetails

vLLM

State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.

In 11 listsDetails

Flowise

Flowise simplifies the creation of applications leveraging large language models (LLMs) by providing a drag-and-drop interface for customizing AI workflows, offering easy installation, Docker support, development tools, and documentation for integrating various functionalities such as…

In 12 listsDetails

TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference…

In 8 listsDetails

shimmy

Python-free Rust inference server with OpenAI API compatibility. Supports GGUF and SafeTensors formats with hot model swap, auto-discovery, and single binary deployment for zero-dependency inference. Apache 2.0 licensed.

In 7 listsDetails

Rapid-MLX

The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Drop-in OpenAI replacement. Works with Claude Code, Cursor, Aider.

In 6 listsDetails

Accelerate

Accelerate abstracts exactly and only the boilerplate code related to multi-GPU/TPU/mixed-precision and leaves the rest of your code unchanged.

In 6 listsDetails

llama.cpp

Ports for inferencing LLaMA in C/C++ running on CPUs, supports alpaca, gpt4all, etc.

In 7 listsDetails