Skip to content

Entry

YYLO Benchmark

Appears in 4 awesome lists

Trusted-host experiment runner for AI coding agents: replay historical tasks, supplied prompts and workflows as independent attempts in fresh repositories, compare models, harnesses and configurations, and evaluate retained outputs with different checks or judges later. (Source Code) MIT

Open github.comyylo-dev/yylo-benchmark

Found in these lists

Awesome Analytics

Section: Developer analytics · Trusted-host experiment runner for AI coding agents: replay historical tasks, supplied prompts and workflows as independent attempts in fresh repositories, compare models, harnesses and configurations, and evaluate retained outputs with different checks or judges later. (Source Code) MIT

FreshScore 89

AWESOME DATA SCIENCE

Section: Tools · Open-source trusted-host experiment runner for historical agent tasks, supplied coding prompts, and workflows. Runs independent attempts with retained outputs, and compares models, harnesses, and configurations with different checks or judges later. MIT licensed.

FreshScore 93

Awesome Open Source AI

Section: 9. Evaluation, Benchmarks & Datasets · Longitudinal evaluation for agent runs: executes task prompts in private fresh-repository attempt workspaces and retains attempt evidence, evaluator provenance, and reports for later re-evaluation. MIT licensed.

FreshScore 87

Best Of Agent Harnesses

Section: On the radar · Longitudinal benchmark for coding-agent runs: sandboxed fresh-repo attempts, deterministic and blinded LLM judging, hash-bound evidence.

FreshScore 80

Jev Social

Read-only Instagram, TikTok and LinkedIn research app where Jev selects bounded operations, the local socai CLI runs them in the user's Chrome, and captured evidence becomes cited reports; TikTok media is optional and explicit-request only.

In 12 listsDetails

DeepEval

The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.

In 10 listsDetails

Dex

Gemini CLI extension with three Agent Skills for analytics engineering: explore a warehouse or DuckDB file read-only (profiles, PII flags, verified joins), author dbt models and semantic layers as reviewable diffs, and detect schema drift. Install with gemini extensions install…

In 11 listsDetails

YYLO

Command-line orchestrator for coding agents, repeatable workflows, and receipt-backed repository changes. Each task freezes the protected target SHA and creates a dedicated branch/worktree behind typed task, validation, merge, and release-readiness boundaries; the merge queue owns risk-based…

In 10 listsDetails

lm-evaluation-harness

EleutherAI — https://github.com/EleutherAI/lm-evaluation-harness — the standard academic harness; first-class decontamination; task YAMLs.

In 8 listsDetails

HuggingFace Datasets

The largest hub of ready-to-use NLP datasets for ML models with fast, easy-to-use and efficient data manipulation tools.

In 6 listsDetails

AutoRAG

RAG AutoML tool for automatically finding optimal RAG pipelines. Evaluates and optimizes retrieval-augmented generation with AutoML-style automation for your own data and use-case. Apache 2.0 licensed.

In 5 listsDetails

Evaluate

Evaluate machine learning models (huggingface). pycm - Multi-class confusion matrix. pandas_ml - Confusion matrix. Plotting learning curve: link. yellowbrick - Learning curve. pyroc - Receiver Operating Characteristic (ROC) curves.

In 5 listsDetails