Skip to content

Entry

OpenAI Evals

Appears in 8 awesome lists

https://github.com/openai/evals — the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)

Open github.comopenai/evals

Found in these lists

Awesome Artificial Intelligence

Section: Evals and reliability · An open-source framework and registry for evaluating language models and systems.

ActiveScore 85

awesome-ChatGPT-repositories

Section: Openai · Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.

FreshScore 87

Awesome Agent Evals

Section: 5a · Eval frameworks & harnesses (code-first test-runners) · https://github.com/openai/evals — the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)

FreshScore 87

Awesome Generative AI

Section: LLM Evaluation · Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

SlowScore 58

Awesome Production Machine Learning

Section: Evaluation and Monitoring · Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.

FreshScore 98

Awesome Prompts

Section: Eval & Testing · Open eval framework and benchmark registry — standardizes LLM performance measurement.

FreshScore 96

awesome-python

Section: Other · Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

FreshScore 79

Awesome Startup

Section: Evaluation and Observability · Framework and registry for benchmarking model behavior.

ActiveScore 81

PydanticAI

June 2026 harness-first redesign built around the Capability primitive: a single composable unit bundling instructions, tools, lifecycle hooks, and model settings. The split between a small stable core and a fast-moving pydantic-ai-harness lets capabilities graduate as they prove essential, while…

In 13 listsDetails

promptfoo

🟢 — LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed — track governance. · updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,…

In 13 listsDetails

FunASR

industrial-grade ASR toolkit; 170× realtime on GPU, 50+ languages, built-in VAD, punctuation, speaker diarization, and emotion detection. Includes non-autoregressive SenseVoice and LLM-based Fun-ASR-Nano models.

In 12 listsDetails

DeepEval

The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.

In 10 listsDetails

Deepchecks

Validation & testing of machine learning models and data during model development, deployment, and production. This includes checks and suites related to various types of issues, such as model performance, data integrity, distribution mismatches, and more.

In 9 listsDetails

AutoGPT

AutoGPT provides accessible AI tools for building and using AI agents, offering a comprehensive framework including Forge for agent creation, agbenchmark for performance evaluation, a leaderboard for competition, a user-friendly UI, and CLI for seamless integration and management github | github…

In 10 listsDetails

lm-evaluation-harness

EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.

In 8 listsDetails

OpenAI Cookbook

OpenAI's cookbook includes examples of prompt engineering.

In 9 listsDetails