Skip to content

Entry

OpenLLMetry

Appears in 8 awesome lists

OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.

Open github.comtraceloop/openllmetry

Found in these lists

Awesome Agent Evals

Section: 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) · https://github.com/traceloop/openllmetry — OSS OTel instrumentation (Py/TS/Go/Ruby) + hosted reliability platform.

FreshScore 87

Awesome Harness Engineering

Section: Observability & Tracing · OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.

FreshScore 93

Awesome LangChain

Section: Platforms · Open-source observability for your LLM application, based on OpenTelemetry

FreshScore 94

Awesome LLMOps

Section: Observability · OpenTelemetry-based observability and monitoring for LLM and agents workflows.

FreshScore 91

Awesome local LLM

Section: Testing, Evaluation and Observability · an open-source observability for your LLM application, based on OpenTelemetry

FreshScore 90

Awesome Open Source AI

Section: 8. MLOps / LLMOps & Production · Open-source observability for GenAI/LLM applications based on OpenTelemetry with 25+ integration backends.

FreshScore 87

Awesome Production Machine Learning

Section: Evaluation and Monitoring · OpenLLMetry provides developers with deep visibility into Large Language Model applications through performance monitoring, execution tracing, and debugging capabilities.

FreshScore 98

awesome-python

Section: Monitoring and Observability · Open-source observability for your GenAI or LLM application, based on OpenTelemetry

FreshScore 79

Opik

Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…

In 18 listsDetails

Phoenix

Open-source AI observability & evaluation platform (Arize) — OpenTelemetry-native tracing for agents, LLM-as-judge evals, versioned datasets & experiments for prompt regression testing, prompt management with version control and replay, plus an MCP endpoint so Claude Code/Cursor can query traces…

In 12 listsDetails

Langfuse

The most widely adopted self-hostable LLM observability platform: traces every agent step, manages prompt versions, and runs evals in one tool. Preferred over cloud-only alternatives when data residency or cost control is a constraint.

In 11 listsDetails

Helicone

Open-source LLM observability proxy (YC W23) with the largest open-source pricing database (300+ models). One-line proxy integration provides cost tracking, token monitoring, session tracing, and prompt versioning across providers. The AI Gateway component handles request routing and caching with…

In 9 listsDetails

Future AGI

🟢🟠 — Self-hostable evaluation and tracing platform with an OpenAI-compatible Go gateway for inline model-traffic checks, MCP/tool permissions and optional external guardrail services. (Future AGI) Caveat: the core is Apache-2.0; enterprise components have a separate commercial license. Gateway…;…

In 9 listsDetails

Maxim AI

The agent simulation, evaluation, and observability platform helping product teams ship their AI applications with the quality and speed needed for real-world use.

In 9 listsDetails

Portkey AI Gateway

🟢🟠 — Open AI gateway with provider routing, fallback and retry controls, guardrail integrations, observability, and MCP traffic support for model and agent applications. (Portkey) — note: the gateway is general infrastructure rather than a standalone security scanner; model-provider…

In 7 listsDetails

logfire

AI observability platform from the Pydantic team with a unique angle: all trace data is SQL-queryable (PostgreSQL-compatible), so coding agents can query production observability data directly via the Logfire MCP server. Full-stack OTEL tracing covers both the AI layer and backend — letting you…

In 5 listsDetails