Awesome Agent Evals
Section: 5a · Eval frameworks & harnesses (code-first test-runners) · EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.
Entry
Appears in 8 awesome lists
EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.
Section: 5a · Eval frameworks & harnesses (code-first test-runners) · EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.
Section: 评估 Evaluation · A framework for few-shot evaluation of language models.
Section: Evaluation and Benchmarks · unified framework for LM benchmark evaluation.
Section: 9. Evaluation, Benchmarks & Datasets · De-facto standard for generative model evaluation.
Section: Evaluation and Monitoring · Language Model Evaluation Harness is a framework to test generative language models on a large number of different evaluation tasks.
Section: Tools & Libraries · EleutherAI's unified LLM evaluation framework
Section: Other · A framework for few-shot evaluation of language models.
Section: LLM 评测与治理 (LLM Evaluation & Harness)
June 2026 harness-first redesign built around the Capability primitive: a single composable unit bundling instructions, tools, lifecycle hooks, and model settings. The split between a small stable core and a fast-moving pydantic-ai-harness lets capabilities graduate as they prove essential, while…
🟢 — LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed — track governance. · updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,…
The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.
Validation & testing of machine learning models and data during model development, deployment, and production. This includes checks and suites related to various types of issues, such as model performance, data integrity, distribution mismatches, and more.
https://github.com/openai/evals — the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)
Evaluate machine learning models (huggingface). pycm - Multi-class confusion matrix. pandas_ml - Confusion matrix. Plotting learning curve: link. yellowbrick - Learning curve. pyroc - Receiver Operating Characteristic (ROC) curves.