Skip to content

Entry

Training language models to follow instructions with human feedback

Appears in 6 awesome lists

This paper presents an RLHF approach to using supervised learning to fine-tuning. It is also known as a paper that illustrates the kernel of ChatGPT's thinking. Presumably, ChatGPT is an extended version of InstructGPT that enables fine-tuning on larger datasets.

Open arxiv.org

Found in these lists

Awesome AI Papers

Section: NLP

SlowScore 53

Awesome Artificial Intelligence

Section: Foundational papers · Established the instruction tuning and RLHF recipe used by InstructGPT.

ActiveScore 85

Awesome GPT-4

Section: Papers · (InstructGPT)

StaleScore 48

awesome-nlp

Section: Instruction Tuning and Preference Optimization · training LMs to follow instructions with human feedback.

FreshScore 91

Awesome Transformer & Transfer Learning in NLP

Section: Papers · by OpenAI. They call the resulting models InstructGPT. ChatGPT is a sibling model to InstructGPT.

StaleScore 52

awesome-chatgpt

Section: The technical principle of ChatGPT · This paper presents an RLHF approach to using supervised learning to fine-tuning. It is also known as a paper that illustrates the kernel of ChatGPT's thinking. Presumably, ChatGPT is an extended version of InstructGPT that enables fine-tuning on larger datasets.

StaleScore 46

ReAct

The foundational paper defining the Thought/Action/Observation loop structure that underlies virtually every agent harness. Required reading for understanding why the loop is structured the way it is and where each harness component maps onto the reasoning-acting cycle.

In 7 listsDetails

Attention Is All You Need

(AIAYN) - Introducing multi-head self-attention neural networks with positional encoding to do sentence-level NLP without any RNN nor CNN - this paper is a must-read (also see this explanation and this visualization of the paper).

In 6 listsDetails

Chain-of-Thought Prompting

foundational result; intermediate reasoning steps improve performance.

In 5 listsDetails

Constitutional AI

Uses written principles and model-generated critiques to scale supervision while retaining explicit behavioral constraints. (arXiv 2022)

In 4 lists

Language Models are Few-Shot Learners

by Tom B. Brown (OpenAI) et al. - "We train GPT-3, an autoregressive language model with 175 billion parameters :scream:, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting."

In 4 lists

Direct Preference Optimization

Reframed preference alignment as a simple classification objective without explicit reward modelling.

In 3 lists

Training Compute-Optimal Large Language Models

by Hoffmann et al. at DeepMind. TLDR: introduces a new 70B LM called "Chinchilla" that outperforms much bigger LMs (GPT-3, Gopher). DeepMind has found the secret to cheaply scale large language models — to be compute-optimal, model size and training data must be scaled equally. It shows that most…

In 3 lists

LoRA

and QLoRA - low-rank adapters and quantized fine-tuning; the standard for adapting LMs to NLP tasks on modest hardware.

In 2 lists