Awesome Artificial Intelligence
Section: Evals and reliability · A practical method for building task suites, graders, transcripts, and evaluation harnesses.
ActiveScore 86
Entry
Demystifying Evals for AI Agents
Appears in 4 awesome lists
Anthropic — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents · blog — Best primary on agent-specific evals: task design, outcome vs trajectory, isolated trials, pass@k vs pass^k.
Found in these lists
Awesome Agent Evals
Section: Must-read starter set (read these first) · Anthropic — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents · blog — Best primary on agent-specific evals: task design, outcome vs trajectory, isolated trials, pass@k vs pass^k.
FreshScore 88
Awesome Forward Deployment Engineering (FDE)
Section: Industry Intelligence (The Gold Standard) · Deep dive into the "Evals" mindset required for production AI.
FreshScore 81
Awesome Harness Engineering
Section: Foundations · Anthropic's framework for evaluating agent behavior: what to measure, how to build eval harnesses, and why unit-test-style evals fail for agents.
FreshScore 93
Often listed next to it
OpenAI Evals
https://github.com/openai/evals — the completion_fn abstraction = swap the system-under-test. (Best-practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices)
In 8 listsDetails
Ragas
Evaluation and experimentation for retrieval and generative AI applications.
In 2 lists
Hidden Technical Debt: Agent Evaluation Infrastructure
Han-Chung Lee — https://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/ · blog — Control/data plane, the five eval surfaces, state deltas. "Chat eval was a spreadsheet; agent eval is a system."
The Second Half
Shunyu Yao — https://ysymyth.github.io/The-Second-Half/ · blog — "Evaluation becomes more important than training." The field-level why.
How to Build Good Language Modeling Benchmarks
Ofir Press — https://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/ · blog — Natural / auto-evaluatable / challenging; the "-200%" difficulty target; ~1-yr saturation.
An LLM-as-Judge Won't Save the Product, Fixing Your Process Will
Eugene Yan — https://eugeneyan.com/writing/eval-process/ · blog — Process over tooling; evals as the scientific method.
Who Validates the Validators? (EvalGen)
Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, Arawjo (UIST '24) — https://arxiv.org/abs/2404.12272 · paper — "Criteria drift": you can't write the rubric before you grade.
LLM Evals FAQ
Hamel Husain & Shreya Shankar — https://hamel.dev/blog/posts/evals-faq/ · blog — The densest operational Q&A: error analysis, binary judgments, the benevolent-dictator labeler.