Entry
Appears in 7 awesome lists
OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.
Section: Observability & Tracing · OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.
Section: Platforms · Open-source observability for your LLM application, based on OpenTelemetry
Section: Observability · OpenTelemetry-based observability and monitoring for LLM and agents workflows.
Section: Testing, Evaluation and Observability · an open-source observability for your LLM application, based on OpenTelemetry
Section: 8. MLOps / LLMOps & Production · Open-source observability for GenAI/LLM applications based on OpenTelemetry with 25+ integration backends.
Section: Evaluation and Monitoring · OpenLLMetry provides developers with deep visibility into Large Language Model applications through performance monitoring, execution tracing, and debugging capabilities.
Section: Monitoring and Observability · Open-source observability for your GenAI or LLM application, based on OpenTelemetry
Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…
LLM engineering platform for model tracing, prompt management, and application evaluation. Langfuse helps teams collaboratively debug, analyze, and iterate on their LLM applications such as chatbots or AI agents. (Demo, Source Code, Clients) MIT Docker
Open-source AI observability & evaluation platform (Arize) — OpenTelemetry-native tracing for agents, LLM-as-judge evals, versioned datasets & experiments for prompt regression testing, prompt management with version control and replay, plus an MCP endpoint so Claude Code/Cursor can query traces…
The most widely adopted self-hostable LLM observability platform: traces every agent step, manages prompt versions, and runs evals in one tool. Preferred over cloud-only alternatives when data residency or cost control is a constraint.
Validation & testing of machine learning models and data during model development, deployment, and production. This includes checks and suites related to various types of issues, such as model performance, data integrity, distribution mismatches, and more.
The agent simulation, evaluation, and observability platform helping product teams ship their AI applications with the quality and speed needed for real-world use.
Interactive reports to analyze machine learning models during validation or production monitoring.
Open-source LLM observability proxy (YC W23) with the largest open-source pricing database (300+ models). One-line proxy integration provides cost tracking, token monitoring, session tracing, and prompt versioning across providers. The AI Gateway component handles request routing and caching with…