Skip to content

Entry

DeepEval

Appears in 10 awesome lists

The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.

Open github.comconfident-ai/deepeval

Found in these lists

awesome-ChatGPT-repositories

Section: NLP · The Evaluation Framework for LLMs

FreshScore 87

Awesome Data Analysis

Section: Tools · Pytest-style unit testing framework for LLMs. Metrics for RAG, agents, hallucination, summarization, and custom criteria.

FreshScore 79

Awesome Agent Evals

Section: 5a · Eval frameworks & harnesses (code-first test-runners) · https://github.com/confident-ai/deepeval — "pytest for LLMs," 40+ metrics (G-Eval, RAG, hallucination) + red-team; ~2M evals/day; hosted cloud. 🆕

FreshScore 87

Awesome Generative AI

Section: LLM Evaluation · The LLM Evaluation Framework

SlowScore 58

Awesome Harness Engineering

Section: Evals & Verification · The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.

FreshScore 93

Awesome LLM Resources

Section: 评估 Evaluation · a simple-to-use, open-source LLM evaluation framework, for evaluating and testing large-language model systems.

FreshScore 91

Awesome Open Source AI

Section: 9. Evaluation, Benchmarks & Datasets · The "Pytest for LLMs".

FreshScore 87

Awesome Production Machine Learning

Section: Evaluation and Monitoring · DeepEval is a simple-to-use, open-source evaluation framework for LLM applications.

FreshScore 98

Awesome Prompts

Section: Eval & Observability · Unit testing for LLMs — G-Eval, hallucination, RAG faithfulness, agentic task metrics.

FreshScore 96

awesome-python

Section: LLM and Inference · The LLM Evaluation Framework

FreshScore 79

LangChain

Langchain integrates various providers like Anthropic, AWS, and OpenAI, and offers tools for components such as LLMs, chat models, and data analysis, supporting functionalities from Alpha Vantage to YouTube github | docs

In 22 listsDetails

Opik

Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and…

In 18 listsDetails

transformers

(formerly known as pytorch-transformers and pytorch-pretrained-bert) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in…

In 14 listsDetails

mem0

The most-deployed memory layer for AI agents — drop-in add/search/update/delete fact storage with LLM-extracted long-term / short-term / semantic memory: automatic fact consolidation (dedup, contradiction resolution, decay) so agents keep "context that persists" without context bloat;…

In 14 listsDetails

LlamaIndex

(MIT) provides modules for structured outputs at different levels of abstraction, including output parsers for text completion endpoints, Pydantic programs for mapping prompts to structured outputs using function calling or output parsing, and pre-defined Pydantic programs for specific output types.

In 14 listsDetails

Dify

February 2026 release making human oversight a native workflow primitive: suspend execution at critical decision points, expose review-and-edit UI mid-flow, and route subsequent execution based on human action (approve/reject/escalate). Demonstrates how HITL transitions from bolt-on approval gates…

In 14 listsDetails

LocalAI

robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video,…

In 14 listsDetails

Whisper

General-purpose speech recognition model for multilingual transcription, translation, language identification, and related speech tasks. Python · Self-host · infra.

In 13 listsDetails