Entry
Appears in 4 awesome lists
Hugging Face β https://github.com/huggingface/lighteval β π all-in-one harness across transformers/vLLM/TGI/nanotron, 1000+ tasks; HF's successor to evaluate.
Section: 5a Β· Eval frameworks & harnesses (code-first test-runners) Β· Hugging Face β https://github.com/huggingface/lighteval β π all-in-one harness across transformers/vLLM/TGI/nanotron, 1000+ tasks; HF's successor to evaluate.
Section: θ―δΌ° Evaluation Β· Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends.
Section: 9. Evaluation, Benchmarks & Datasets Β· Evaluation toolkit for LLMs across multiple backends with reusable tasks, metrics, and result tracking.
Section: Evaluation and Monitoring Β· LightEval is a lightweight LLM evaluation suite.
June 2026 harness-first redesign built around the Capability primitive: a single composable unit bundling instructions, tools, lifecycle hooks, and model settings. The split between a small stable core and a fast-moving pydantic-ai-harness lets capabilities graduate as they prove essential, whileβ¦
π’ β LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed β track governance. Β· updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,β¦
The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.
Validation & testing of machine learning models and data during model development, deployment, and production. This includes checks and suites related to various types of issues, such as model performance, data integrity, distribution mismatches, and more.
EleutherAI β https://github.com/EleutherAI/lm-evaluation-harness β the standard academic harness; first-class decontamination; task YAMLs.
https://github.com/openai/evals β the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)
The largest hub of ready-to-use NLP datasets for ML models with fast, easy-to-use and efficient data manipulation tools.