Awesome LLMOps
Section: Experiment Tracking · Sacred is a tool to help you configure, organize, log and reproduce experiments.
Entry
Appears in 6 awesome lists
Python tool to help you configure, organize, log and reproduce experiments. Like a notebook lab in the context of Chemistry/Biology. The community has built multiple add-ons leveraging the proposed standard.
Section: Experiment Tracking · Sacred is a tool to help you configure, organize, log and reproduce experiments.
Section: Tools · Python tool to help you configure, organize, log and reproduce experiments. Like a notebook lab in the context of Chemistry/Biology. The community has built multiple add-ons leveraging the proposed standard.
Section: Model Lifecycle · A tool to help you configure, organize, log and reproduce experiments.
Section: Model, Data and Experiment Management · Tool to help you configure, organize, log and reproduce machine learning experiments.
Section: Machine Learning Frameworks · Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.
Section: Experimentation · A tool to help you configure, organize, log, and reproduce experiments.
Transpile trained ML models into other languages. sklearn-porter - Transpile trained scikit-learn estimators to C, Java, JavaScript and others. mlflow - Manage the machine learning lifecycle, including experimentation, reproducibility and deployment. skll - Command-line utilities to make it easier…
🟢 — LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed — track governance. · updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,…
All-in-one web-based IDE for machine learning and data science. The workspace is deployed as a docker container and is preloaded with a variety of popular data science libraries (e.g., Tensorflow, PyTorch) and dev tools (e.g., Jupyter, VS Code).
Praison AI is a low-code, centralized framework leveraging AutoGen and CrewAI to simplify creating and orchestrating multi-agent systems for LLM applications, emphasizing customization and ease of human-agent interaction github | demo | website
AI-native database built for LLM applications with incredibly fast hybrid search of dense vector, sparse vector, tensor (multi-vector), and full-text. Powers RAGFlow's document engine. Apache 2.0 licensed.
Self-hosted ML experiment tracker designed to handle 10,000s of training runs with performant UI and SDK for programmatic access. Apache 2.0 licensed.