Awesome C++
Section: Machine Learning · A high-performance neural network inference computing framework optimized for mobile platforms. [BSD]
Entry
Appears in 6 awesome lists
High-performance neural network inference framework optimized for mobile platforms. No third-party dependencies, cross-platform, and runs faster than all known open-source frameworks on mobile CPU. Powers Tencent apps including QQ, WeChat, and Pitu. BSD-3-Clause licensed.
Section: Machine Learning · A high-performance neural network inference computing framework optimized for mobile platforms. [BSD]
Section: Optimizations · ncnn is a high-performance neural network inference framework optimized for the mobile platform.
Section: Inference and hardware · Mobile neural network inference, including INT8 deployment.
Section: 11. Specialized Domains · High-performance neural network inference framework optimized for mobile platforms. No third-party dependencies, cross-platform, and runs faster than all known open-source frameworks on mobile CPU. Powers Tencent apps including QQ, WeChat, and Pitu. BSD-3-Clause licensed.
Section: Libraries · High-performance neural network inference framework with Vulkan based GPU inference. [BSD 3-clause]
Section: Other · ncnn is a high-performance neural network inference framework optimized for the mobile platform
Transpile trained ML models into other languages. sklearn-porter - Transpile trained scikit-learn estimators to C, Java, JavaScript and others. mlflow - Manage the machine learning lifecycle, including experimentation, reproducibility and deployment. skll - Command-line utilities to make it easier…
Pure C/C++ inference engine with GGUF format support. The gold standard for CPU/GPU/Apple Silicon on-device running. Includes llama-server for OpenAI-compatible API. Now at 100K+ stars.
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Flink and DataFlow. [Apache2]
State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.
General purpose gradient boosting on decision trees library with categorical features support out of the box. It is easy to install, contains fast inference implementation and supports CPU and GPU (even multi-GPU) computation.
is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete…
is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berkeley Vision and Learning Center (BVLC) and community contributors.
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference…