Awesome Artificial Intelligence
Section: Evals and reliability · An open-source framework and registry for evaluating language models and systems.
Entry
Appears in 8 awesome lists
https://github.com/openai/evals — the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)
Section: Evals and reliability · An open-source framework and registry for evaluating language models and systems.
Section: Openai · Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.
Section: 5a · Eval frameworks & harnesses (code-first test-runners) · https://github.com/openai/evals — the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)
Section: LLM Evaluation · Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Section: Evaluation and Monitoring · Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.
Section: Eval & Testing · Open eval framework and benchmark registry — standardizes LLM performance measurement.
Section: Other · Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Section: Evaluation and Observability · Framework and registry for benchmarking model behavior.
June 2026 harness-first redesign built around the Capability primitive: a single composable unit bundling instructions, tools, lifecycle hooks, and model settings. The split between a small stable core and a fast-moving pydantic-ai-harness lets capabilities graduate as they prove essential, while…
🟢 — LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed — track governance. · updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,…
industrial-grade ASR toolkit; 170× realtime on GPU, 50+ languages, built-in VAD, punctuation, speaker diarization, and emotion detection. Includes non-autoregressive SenseVoice and LLM-based Fun-ASR-Nano models.
The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.
Validation & testing of machine learning models and data during model development, deployment, and production. This includes checks and suites related to various types of issues, such as model performance, data integrity, distribution mismatches, and more.
AutoGPT provides accessible AI tools for building and using AI agents, offering a comprehensive framework including Forge for agent creation, agbenchmark for performance evaluation, a leaderboard for competition, a user-friendly UI, and CLI for seamless integration and management github | github…
EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.
OpenAI's cookbook includes examples of prompt engineering.