Skip to content

Entry

LangKit

Appears in 6 awesome lists

LLM monitoring toolkit from WhyLabs that extracts safety and security signals including jailbreak similarity, prompt-injection similarity, hallucination checks, PII patterns, toxicity, and refusal metrics for production observability.

Open github.comwhylabs/langkit

Found in these lists

Awesome AI Security Tools

Section: Scanners, Evals & Guardrails · 🟢 — LLM monitoring toolkit extracting safety/security signals such as jailbreak similarity, prompt-injection similarity, hallucination checks, PII patterns, toxicity, and refusal metrics. (WhyLabs) · updated 2024-11-22)

FreshScore 91

Awesome Agent Evals

Section: 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) · https://github.com/whylabs/langkit — high-throughput text-signal metrics (toxicity, PII, jailbreak) for production monitoring.

FreshScore 87

Awesome LLM Resources

Section: 数据 Data · An open-source toolkit for monitoring Large Language Models (LLMs). Extracts signals from prompts & responses, ensuring safety & security.

FreshScore 91

Awesome LLMOps

Section: LLMOps · Out-of-the-box LLM telemetry collection library that extracts features and profiles prompts, responses and metadata about how your LLM is performing over time to find problems at scale.

FreshScore 91

Hacking Tools

Section: AI & Machine Learning Security · LLM monitoring toolkit from WhyLabs that extracts safety and security signals including jailbreak similarity, prompt-injection similarity, hallucination checks, PII patterns, toxicity, and refusal metrics for production observability.

FreshScore 90

Awesome AI Agents: Tools, Resources, and Projects

Section: Tools · 🔍 LangKit - An open-source toolkit for monitoring Large Language Models (LLMs). 📚 Extracts signals from prompts & responses, ensuring safe…

SlowScore 66

Opik

Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…

In 18 listsDetails

promptfoo

🟢 — LLM eval + red-teaming/pentesting CLI with 50+ attack plugins (MIT). Note: OpenAI announced an acquisition agreement in March 2026; remains MIT-licensed — track governance. · updated 2026-10-13); Related: Promptfoo CI Action (checkout evaluation, not base/head diff; defaults to latest,…

In 13 listsDetails

Phoenix

Open-source AI observability & evaluation platform (Arize) — OpenTelemetry-native tracing for agents, LLM-as-judge evals, versioned datasets & experiments for prompt regression testing, prompt management with version control and replay, plus an MCP endpoint so Claude Code/Cursor can query traces…

In 12 listsDetails

Langfuse

The most widely adopted self-hostable LLM observability platform: traces every agent step, manages prompt versions, and runs evals in one tool. Preferred over cloud-only alternatives when data residency or cost control is a constraint.

In 11 listsDetails

MarkItDown

Python tool for converting files and office documents to Markdown. Supports PDF, PowerPoint, Word, Excel, images, audio, HTML, and more with OCR and transcription capabilities. MIT licensed.

In 9 listsDetails

Helicone

Open-source LLM observability proxy (YC W23) with the largest open-source pricing database (300+ models). One-line proxy integration provides cost tracking, token monitoring, session tracing, and prompt versioning across providers. The AI Gateway component handles request routing and caching with…

In 9 listsDetails

OpenLLMetry

OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.

In 8 listsDetails

Maxim AI

The agent simulation, evaluation, and observability platform helping product teams ship their AI applications with the quality and speed needed for real-world use.

In 9 listsDetails