Awesome Agent Evals
Section: 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) · https://github.com/traceloop/openllmetry — OSS OTel instrumentation (Py/TS/Go/Ruby) + hosted reliability platform.
Entry
Appears in 8 awesome lists
OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.
Section: 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) · https://github.com/traceloop/openllmetry — OSS OTel instrumentation (Py/TS/Go/Ruby) + hosted reliability platform.
Section: Observability & Tracing · OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.
Section: Platforms · Open-source observability for your LLM application, based on OpenTelemetry
Section: Observability · OpenTelemetry-based observability and monitoring for LLM and agents workflows.
Section: Testing, Evaluation and Observability · an open-source observability for your LLM application, based on OpenTelemetry
Section: 8. MLOps / LLMOps & Production · Open-source observability for GenAI/LLM applications based on OpenTelemetry with 25+ integration backends.
Section: Evaluation and Monitoring · OpenLLMetry provides developers with deep visibility into Large Language Model applications through performance monitoring, execution tracing, and debugging capabilities.
Section: Monitoring and Observability · Open-source observability for your GenAI or LLM application, based on OpenTelemetry
Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…
Open-source AI observability & evaluation platform (Arize) — OpenTelemetry-native tracing for agents, LLM-as-judge evals, versioned datasets & experiments for prompt regression testing, prompt management with version control and replay, plus an MCP endpoint so Claude Code/Cursor can query traces…
The most widely adopted self-hostable LLM observability platform: traces every agent step, manages prompt versions, and runs evals in one tool. Preferred over cloud-only alternatives when data residency or cost control is a constraint.
Open-source LLM observability proxy (YC W23) with the largest open-source pricing database (300+ models). One-line proxy integration provides cost tracking, token monitoring, session tracing, and prompt versioning across providers. The AI Gateway component handles request routing and caching with…
🟢🟠 — Self-hostable evaluation and tracing platform with an OpenAI-compatible Go gateway for inline model-traffic checks, MCP/tool permissions and optional external guardrail services. (Future AGI) Caveat: the core is Apache-2.0; enterprise components have a separate commercial license. Gateway…;…
The agent simulation, evaluation, and observability platform helping product teams ship their AI applications with the quality and speed needed for real-world use.
🟢🟠 — Open AI gateway with provider routing, fallback and retry controls, guardrail integrations, observability, and MCP traffic support for model and agent applications. (Portkey) — note: the gateway is general infrastructure rather than a standalone security scanner; model-provider…
AI observability platform from the Pydantic team with a unique angle: all trace data is SQL-queryable (PostgreSQL-compatible), so coding agents can query production observability data directly via the Logfire MCP server. Full-stack OTEL tracing covers both the AI layer and backend — letting you…