Skip to content
56

Awesome Code LLM

👨‍💻 An awesome and curated list of best code-LLM for research.

1.3k stars76 forks50 entriesLast push Dec 10, 2024 (1 year ago)License MIT

This page lists names, links and short descriptions. The original list on GitHub is the source and belongs to its authors.

Evaluation Toolkit:

bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.

In 2 lists

code-eval

A framework for the evaluation of autoregressive code generation language models on HumanEval.

SandboxFusion

A secure sandbox for running and judging code generated by LLMs.

Awesome Code LLMs Leaderboard

Evalperf Leaderboard

Evaluating LLMs for Efficient Code Generation.

Aider Code Editing Leaderboard

Measuring the LLM’s coding ability, and whether it can write new code that integrates into existing code.

BigCodeBench Leaderboard

BigCodeBench evaluates LLMs with practical and challenging programming tasks.

LiveCodeBench Leaderboard

Holistic and Contamination Free Evaluation of Large Language Models for Code.

Big Code Models Leaderboard

Compare performance of base multilingual code generation models on HumanEval benchmark and MultiPL-E.

BIRD Leaderboard

BIRD contains over 12,751 unique question-SQL pairs, 95 big databases with a total size of 33.4 GB. It also covers more than 37 professional domains, such as blockchain, hockey, healthcare and education, etc.

CanAiCode Leaderboard

CanAiCode Leaderboard

Link may be dead

Coding LLMs Leaderboard

Coding LLMs Leaderboard

CRUXEval Leaderboard

CRUXEval is a benchmark complementary to HumanEval and MBPP measuring code reasoning, understanding, and execution capabilities!

EvalPlus Leaderboard

EvalPlus evaluates AI Coders with rigorous tests.

InfiBench Leaderboard

InfiBench is a comprehensive benchmark for code large language models evaluating model ability on answering freeform real-world questions in the code domain.

InterCode Leaderboard

InterCode is a benchmark for evaluating language models on the interactive coding task. Given a natural language request, an agent is asked to interact with a software system (e.g., database, terminal) with code to resolve the issue.

Program Synthesis Models Leaderboard

They created this leaderboard to help researchers easily identify the best open-source model with an intuitive leadership quadrant graph. They evaluate the performance of open-source code models to rank them based on their capabilities and market adoption.

Link may be dead

Spider Leaderboard

Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

Awesome Code LLMs Papers >Awesome Code Pre-Training Papers

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

Preprint

Qwen2.5-Coder Technical Report

Preprint

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

Preprint

StarCoder 2 and The Stack v2: The Next Generation

Preprint

DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

Preprint

Code Llama: Open Foundation Models for Code

Preprint

In 2 lists

Textbooks Are All You Need

Preprint

In 2 lists

CodeT5+: Open Code Large Language Models for Code Understanding and Generation

Preprint

StarCoder: may the source be with you!

Preprint

CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

ICLR23

CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X

Preprint

SantaCoder: don't reach for the stars!

Preprint

CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

ICLR'23

Evaluating Large Language Models Trained on Code

Preprint

In 2 lists

Awesome Code LLMs Papers >Awesome Code Instruction-Tuning Papers

Magicoder: Source Code Is All You Need

ICML'24

OctoPack: Instruction Tuning Code Large Language Models

ICLR'24

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

Preprint

Code Alpaca: An Instruction-following LLaMA Model trained on code generation instructions

Preprint

Awesome Code LLMs Papers >Awesome Code Alignment Papers

ProSec: Fortifying Code LLMs with Proactive Security Alignment

Preprint

PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models

Preprint

PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

Preprint

RLTF: Reinforcement Learning from Unit Test Feedback

Preprint

Execution-based Code Generation using Deep Reinforcement Learning

TMLR'23

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

NeurIPS'22

Awesome Code LLMs Papers >Awesome Code Prompting Papers

From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging

Preprint

Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs

AAAI'25

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step

ACL'24

SelfEvolve: A Code Evolution Framework via Large Language Models

Preprint

Demystifying GPT Self-Repair for Code Generation

ICLR'24

Teaching Large Language Models to Self-Debug

ICLR'24

In 2 lists

LEVER: Learning to Verify Language-to-Code Generation with Execution

ICML'23

Coder Reviewer Reranking for Code Generation

ICML'23

CodeT: Code Generation with Generated Tests

ICLR'23

See category
99

Awesome DeepSeek Harness (DSH) Plugin

awesome-dsh-plugin/awesome-dsh-plugin

A curated list of plugins for DeepSeek Harness (dsh) · DeepSeek Harness 插件精选列表

Fresh★ 25k1416 entriesPushed today
98

Awesome OpenClaw Skills

VoltAgent/awesome-openclaw-skills

The awesome collection of OpenClaw skills. 5,400+ skills filtered and categorized from the official OpenClaw Skills Registry.🦞

Fresh★ 54k843 entriesPushed 16 days ago
96

Awesome Prompts

ai-boost/awesome-prompts

Curated prompts, frameworks and papers for LLMs and agents, with an engineering bias. Prompt engineering, context engineering, agent harnesses, red teaming.

Fresh★ 9.3k365 entriesPushed today
96

Awesome Open Source Alternatives

piotrkulpinski/open-source-alternatives

Curated list of open source alternatives to proprietary software.

Fresh★ 6.9k860 entriesPushed today
95

Awesome README

matiassingers/awesome-readme

A curated list of awesome READMEs

Fresh★ 22k144 entriesPushed 5 days ago
94

Awesome Persona Distill Skills

xixu-me/awesome-persona-distill-skills

Curated list of Agent Skills centered on people, relationships, commemorative scenes, and methodological perspectives

Fresh★ 4.7k72 entriesPushed today