Skip to content

Entry

lm-evaluation-harness

Appears in 8 awesome lists

EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.

Open github.comeleutherai/lm-evaluation-harness

Found in these lists

Awesome Agent Evals

Section: 5a · Eval frameworks & harnesses (code-first test-runners) · EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.

FreshScore 87

Awesome LLM Resources

Section: 评估 Evaluation · A framework for few-shot evaluation of language models.

FreshScore 91

awesome-nlp

Section: Evaluation and Benchmarks · unified framework for LM benchmark evaluation.

FreshScore 91

Awesome Open Source AI

Section: 9. Evaluation, Benchmarks & Datasets · De-facto standard for generative model evaluation.

FreshScore 87

Awesome Production Machine Learning

Section: Evaluation and Monitoring · Language Model Evaluation Harness is a framework to test generative language models on a large number of different evaluation tasks.

FreshScore 98

Awesome Prompts

Section: Tools & Libraries · EleutherAI's unified LLM evaluation framework

FreshScore 96

awesome-python

Section: Other · A framework for few-shot evaluation of language models.

FreshScore 79

Indie Hacker Tools Plus

Section: LLM 评测与治理 (LLM Evaluation & Harness)

FreshScore 89

PydanticAI

June 2026 harness-first redesign built around the Capability primitive: a single composable unit bundling instructions, tools, lifecycle hooks, and model settings. The split between a small stable core and a fast-moving pydantic-ai-harness lets capabilities graduate as they prove essential, while…

In 13 listsDetails

promptfoo

🟢 — LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed — track governance. · updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,…

In 13 listsDetails

DeepEval

The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.

In 10 listsDetails

Deepchecks

Validation & testing of machine learning models and data during model development, deployment, and production. This includes checks and suites related to various types of issues, such as model performance, data integrity, distribution mismatches, and more.

In 9 listsDetails

OpenAI Evals

https://github.com/openai/evals — the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)

In 8 listsDetails

Evaluate

Evaluate machine learning models (huggingface). pycm - Multi-class confusion matrix. pandas_ml - Confusion matrix. Plotting learning curve: link. yellowbrick - Learning curve. pyroc - Receiver Operating Characteristic (ROC) curves.

In 5 listsDetails

Giskard

🟢 — Open-source evaluation, testing, and red-teaming framework for LLM agents, including agent vulnerability scanning and RAG evaluation workflows. (Giskard AI) · updated 2026-10-09); Related: Moonshot · promptfoo

In 4 listsDetails

Harbor

harbor-framework (Laude Institute / Stanford) — https://github.com/harbor-framework/harbor — 🆕 framework for running agent evals + creating/using RL environments; powers Terminal-Bench 2.0. ~3.7k . ⚠️ name overloaded (cf. av/harbor local-LLM toolkit).

In 4 listsDetails