Skip to content

Entry

Harbor

Appears in 4 awesome lists

harbor-framework (Laude Institute / Stanford) — https://github.com/harbor-framework/harbor — 🆕 framework for running agent evals + creating/using RL environments; powers Terminal-Bench 2.0. ~3.7k . ⚠️ name overloaded (cf. av/harbor local-LLM toolkit).

Open github.comharbor-framework/harbor

Found in these lists

Awesome Agent Evals

Section: 5a · Eval frameworks & harnesses (code-first test-runners) · harbor-framework (Laude Institute / Stanford) — https://github.com/harbor-framework/harbor — 🆕 framework for running agent evals + creating/using RL environments; powers Terminal-Bench 2.0. ~3.7k . ⚠️ name overloaded (cf. av/harbor local-LLM toolkit).

FreshScore 87

Awesome Open Source AI

Section: 9. Evaluation, Benchmarks & Datasets · Framework for running agent evaluations and creating/using RL environments. Evaluate arbitrary agents like Claude Code, OpenHands, and Codex CLI. Build and share benchmarks and environments. Apache 2.0 licensed.

FreshScore 87

Awesome Production Machine Learning

Section: Evaluation and Monitoring · Harbor is a framework for evaluating and optimizing agents and language models, supporting parallel experiments across container environments with built-in benchmark and environment management.

FreshScore 98

Awesome Harness Engineering

Section: Sandboxes & Execution Infrastructure · A generalized harness for evaluating and improving agents at scale, released alongside Terminal-Bench 2.0.

FreshScore 93

Opik

Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…

In 18 listsDetails

PydanticAI

June 2026 harness-first redesign built around the Capability primitive: a single composable unit bundling instructions, tools, lifecycle hooks, and model settings. The split between a small stable core and a fast-moving pydantic-ai-harness lets capabilities graduate as they prove essential, while…

In 13 listsDetails

promptfoo

🟢 — LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed — track governance. · updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,…

In 13 listsDetails

Phoenix

Open-source AI observability & evaluation platform (Arize) — OpenTelemetry-native tracing for agents, LLM-as-judge evals, versioned datasets & experiments for prompt regression testing, prompt management with version control and replay, plus an MCP endpoint so Claude Code/Cursor can query traces…

In 12 listsDetails

Langfuse

The most widely adopted self-hostable LLM observability platform: traces every agent step, manages prompt versions, and runs evals in one tool. Preferred over cloud-only alternatives when data residency or cost control is a constraint.

In 11 listsDetails

DeepEval

The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.

In 10 listsDetails

Deepchecks

Validation & testing of machine learning models and data during model development, deployment, and production. This includes checks and suites related to various types of issues, such as model performance, data integrity, distribution mismatches, and more.

In 9 listsDetails

Helicone

Open-source LLM observability proxy (YC W23) with the largest open-source pricing database (300+ models). One-line proxy integration provides cost tracking, token monitoring, session tracing, and prompt versioning across providers. The AI Gateway component handles request routing and caching with…

In 9 listsDetails