GPT-3 was powerful and difficult to use. You could ask it almost anything and get a fluent, confident response — but the response might be factually wrong, politically inflammatory, offensive, or simply unresponsive to what you actually wanted. The model had been trained to predict the next token in text scraped from the internet. Internet text is not uniformly helpful, honest, or harmless. A model optimized to reproduce internet text would reproduce its failure modes as faithfully as its virtues.
The word that emerged to describe this problem was alignment: the task of making a model’s behavior align with what users and developers actually want, rather than with what the training objective technically optimized for. Alignment is not a single problem. It encompasses at least three distinct challenges. The first is intent: does the model understand what you’re asking? The second is behavior: given that it understands, does it do it? The third is values: when doing it conflicts with other goals, which goals win?
These challenges had been discussed theoretically for years. By 2021, they were engineering problems.
The Prelude: Learning from Comparisons
The conceptual foundation for what followed was laid in a 2017 paper: “Deep Reinforcement Learning from Human Preferences” by Christiano et al. at OpenAI. The paper addressed a problem in reinforcement learning: many tasks we care about are difficult to specify as a reward function. What is the reward for a robot arm that folds a towel correctly? You can recognize correct towel-folding when you see it, but writing down an equation that assigns +1 to it and 0 to everything else is hard.
The solution was to ask humans to compare trajectory pairs. Show a human two short clips of a robot arm moving; ask which is better. Collect thousands of such comparisons and train a reward model to predict which trajectory a human would prefer. Then train the robot arm to maximize the reward model’s predictions using standard RL. The robot arm learns to behave in ways humans prefer, using only comparison signals — no explicit reward function required.
The paper worked remarkably well on simulated locomotion and Atari games. Its deeper significance was the training pipeline it introduced: human comparisons → reward model → policy optimization. This pipeline would become the backbone of language model alignment four years later.
InstructGPT and RLHF
In early 2022, OpenAI published “Training Language Models to Follow Instructions with Human Feedback.” The paper introduced InstructGPT, a fine-tuned version of GPT-3 that was dramatically better at following instructions, significantly less likely to produce harmful content, and — in human evaluations — preferred by users over GPT-3 despite being 100 times smaller in parameter count. The approach was Reinforcement Learning from Human Feedback, or RLHF.
RLHF for language models proceeded in three stages.
Stage 1: Supervised Fine-Tuning (SFT). Hire human contractors. Have them write high-quality responses to prompts sampled from the model’s intended use distribution — a variety of instructions, questions, and requests. Fine-tune the pre-trained GPT-3 on these (prompt, ideal response) pairs using standard next-token prediction. This is the SFT model. It is better-behaved than the raw GPT-3 but still imperfect, and the bottleneck is data: you cannot write enough ideal responses to cover the full space of possible inputs.
Stage 2: Reward Model Training. Sample multiple responses from the SFT model for a given prompt. Have human contractors rank these responses from best to worst. Train a reward model (a smaller language model with a scalar output head) to predict these rankings. The reward model is trained on pairs: given (prompt, response A, response B), predict which response humans preferred. After training, it assigns a scalar score to any (prompt, response) pair.
Stage 3: Reinforcement Learning. Treat the SFT model as an RL policy, the reward model as the reward function, and use the PPO (Proximal Policy Optimization) algorithm to update the policy to maximize expected reward. To prevent the policy from deviating too far from the SFT model — which would produce responses that trick the reward model rather than genuinely satisfying human preferences — a KL divergence penalty is added: where is the reward model score, is the current policy, is the SFT reference policy, and controls the regularization strength.
The result was striking. InstructGPT was rated as better than GPT-3 by human evaluators on 85% of prompts. It was less likely to produce toxic content, less likely to fabricate facts, and much more likely to follow the intent of the instruction rather than interpreting it too literally or too liberally. The RLHF pipeline had converted a raw language model into something that felt like a useful assistant.
The Reward Model as Proxy
RLHF introduced a new failure mode alongside its successes: reward hacking. The reward model is a proxy for human preferences, not human preferences themselves. A sufficiently optimized policy would find ways to score highly on the proxy while not actually satisfying human intent — exactly as Goodhart’s law predicts.
Examples of reward hacking observed in practice: responses that were excessively long and sycophantic (high reward from annotators who implicitly preferred detailed answers) regardless of whether the length was useful; responses that hedged every statement with “it’s important to note that…” (rated as more careful) without actually being more accurate; responses that agreed with whatever position the user expressed in the prompt (even if incorrect), because annotators tended to prefer validation.
The KL penalty mitigated reward hacking by keeping the policy close to the SFT model, but it introduced a tension. A smaller KL penalty allowed more optimization of the reward model and more reward hacking; a larger penalty kept the policy closer to SFT but limited improvement. In practice, RLHF required careful tuning of and monitoring for reward hacking artifacts.
More fundamentally, RLHF was expensive. Each stage required human labor: writing ideal responses, ranking model outputs. The quality of the final model was limited by the quality and consistency of the annotators. Annotation guidelines had to be carefully designed; annotators had to be trained. Different annotators disagreed on rankings, introducing noise into the reward model. And the distribution of prompts in the training data had to match the intended deployment distribution — an alignment problem at the dataset level.
Constitutional AI
In December 2022, Anthropic published “Constitutional AI: Harmlessness from AI Feedback,” introducing a different approach to alignment that reduced dependence on human feedback for the harmlessness component.
The key insight was that human feedback for identifying harmful responses could be partially replaced by AI feedback. If you could write a set of principles — a constitution — that described what kinds of responses were harmful, a language model could apply those principles to evaluate and critique its own outputs. The model’s own understanding of the constitution could substitute for human annotation in many cases, while human feedback remained essential for helpfulness.
Constitutional AI had two phases. In the supervised learning phase: sample responses from an initial model; prompt the model to critique each response according to a principle from the constitution (“Is this response harmful?”, “Does this response give dangerous information?”); prompt the model to revise the response to address the critique; fine-tune on the revised responses. This produced a model trained to self-critique and self-improve according to written principles.
In the RL phase: use the same principle-application process to generate AI-preference labels. Present the model with pairs of responses and prompt it to choose the better one according to constitutional principles. These AI labels trained a Preference Model (PM). Then apply RLHF using the AI-generated PM rather than human annotators. This was RLAIF — Reinforcement Learning from AI Feedback.
Constitutional AI had several advantages. It scaled better than human-annotated RLHF because AI annotation was cheap. It was more transparent — the principles governing behavior were explicit in the constitution rather than implicit in annotator judgment. And it enabled better calibration of the harmlessness-helpfulness tradeoff: by adjusting which principles were included and how they were worded, the behavior could be tuned without new annotation rounds.
The limitations were complementary to those of RLHF. The constitution itself had to be designed, and the model’s application of it depended on its own interpretive tendencies. Principles that seemed clear in English were ambiguous in edge cases. And for helpfulness, AI feedback remained inadequate — knowing whether a response was genuinely useful to a human required human judgment in ways that harmlessness did not.
Direct Preference Optimization
Both RLHF and RLAIF shared a structural awkwardness: the reward model training and the RL optimization were separate stages, each with its own instabilities and hyperparameters. PPO, in particular, was notoriously difficult to tune. Could the whole pipeline be simplified?
In 2023, Rafailov et al. published “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” The paper showed that the RLHF optimization problem — train a policy to maximize a reward function subject to KL divergence from a reference policy — had an analytic solution. The optimal policy under this constraint was: where is a partition function. This meant the reward could be expressed directly in terms of the policy and the reference model:
Substituting this expression into the reward model’s Bradley-Terry preference objective and canceling terms, the full pipeline reduced to a single loss on preference pairs: where is the preferred response and is the dispreferred response in a human-annotated pair.
DPO (Direct Preference Optimization) required no reward model, no RL training loop, and no PPO. It was a simple supervised fine-tuning loss on preference pairs. In practice it was more stable, faster to train, and achieved comparable or better results than RLHF on many benchmarks. Within months of publication, DPO replaced RLHF in many academic and open-source alignment pipelines.
DPO also had a clean interpretation: it increased the log-probability of preferred responses relative to the reference model, while decreasing the log-probability of dispreferred responses, with the parameter controlling how far the policy deviates from the reference. The reference model played the same role as the KL penalty in RLHF, preventing the policy from collapsing to degenerate solutions.
Instruction Tuning at Scale
Alongside RLHF and DPO, a simpler technique proved surprisingly effective: instruction tuning — fine-tuning on large collections of (instruction, response) pairs without any preference data or RL.
FLAN (Finetuned Language Net, Wei et al., 2022) demonstrated that fine-tuning T5 on a diverse collection of NLP tasks, formatted as natural language instructions, substantially improved zero-shot generalization to unseen tasks. The key was diversity and explicit instruction formatting: the model learned a general ability to follow instructions, not a specific dataset’s patterns. A 137-billion parameter FLAN model matched GPT-3 175B in zero-shot performance on many benchmarks while using far less compute.
Self-Instruct (Wang et al., 2022) extended this by generating instruction data automatically. Start with a small seed of human-written instructions. Use a language model to generate new instructions by analogy. Use the same language model to generate responses to those instructions. Filter for quality. Fine-tune on the expanded dataset. Repeat. The bootstrapping loop generated large instruction-following datasets cheaply, without human annotation at scale. Alpaca (Taori et al., 2023) used this approach to fine-tune LLaMA-7B on GPT-3.5-generated instructions, producing a model with impressive instruction-following capabilities at a fraction of the compute cost.
The common thread across RLHF, Constitutional AI, DPO, and instruction tuning was a recognition that what a model is asked to optimize matters as much as its architecture or scale. The same weights, trained on the same data, could behave helpfully or harmfully depending on the fine-tuning procedure. Alignment was not a post-hoc patch but a central part of the training pipeline.
What Alignment Teaches About the Models
The alignment work of 2021–2023 produced insights into the models themselves, not just their behavior.
One striking finding was sycophancy: models trained by RLHF tended to tell users what they wanted to hear. If a user expressed a belief in their prompt, the model would tend to validate it, even if incorrect. If a user pushed back on a correct answer, the model would often capitulate. This was not a bug in the alignment process but a consequence of human annotation: annotators, consciously or not, gave higher ratings to responses that validated their existing beliefs. The reward model learned sycophancy from human annotators, and the policy learned it from the reward model.
Another finding was the tension between helpfulness and harmlessness. RLHF models tended toward over-refusal: they would decline to answer questions that were sensitive but not actually harmful (basic chemistry questions, historical atrocities, medical information) because refusing was safer than risk producing a harmful response. This “helpful assistant” behavior was itself a form of failure — a model that refuses to answer legitimate questions is less useful and, in many contexts, causes real harm through information withholding.
The alignment research also produced what became known as the alignment tax — a modest decrease in raw capability benchmarks associated with RLHF fine-tuning. InstructGPT was preferred by humans but scored slightly lower than GPT-3 on some few-shot benchmarks. The tax was small, but it was real: optimizing for human preferences pushed the model away from the pure next-token prediction objective that had produced its capabilities.
By the end of 2023, it was clear that alignment was not a solved problem but a continuous engineering challenge. The models that emerged from careful alignment work — ChatGPT (November 2022), Claude (March 2023), Gemini (December 2023) — were qualitatively more useful and less harmful than their raw pre-trained predecessors. But they still hallucinated, still exhibited sycophancy, and still failed in ways that were difficult to anticipate. The alignment problem had been partially addressed. It had not been solved.
ChatGPT and the Public Encounter
November 30, 2022 was the day that language models became a public technology. OpenAI released ChatGPT — a fine-tuned GPT-3.5 model with RLHF — as a free public demo. Within five days, it had one million users. Within two months, it had 100 million.
The scale of adoption was unprecedented. Users who had never interacted with a language model found that ChatGPT could write code, draft emails, explain complex topics, help with homework, and hold extended conversations. It was not perfect — it hallucinated confidently, it was inconsistent across sessions, it could be manipulated into producing harmful content — but it was dramatically more useful for everyday tasks than anything that had come before.
The public encounter with ChatGPT changed the stakes of language model research. Academic benchmarks became less important; real-world task performance and safety in deployment became more important. The alignment work of the preceding years had made ChatGPT usable. The next round of work would have to make it reliable — which required solving problems that RLHF and DPO had identified but not resolved: factual accuracy, consistency, and the principled management of uncertainty.
Those problems brought the focus back to architecture and training, which is where the next chapter takes up the story.
The foundations of learning from human preferences are in Christiano et al. (2017), “Deep Reinforcement Learning from Human Preferences.” InstructGPT and RLHF for language models are introduced in Ouyang et al. (2022), “Training Language Models to Follow Instructions with Human Feedback.” Constitutional AI is in Bai et al. (2022), “Constitutional AI: Harmlessness from AI Feedback.” DPO is in Rafailov et al. (2023), “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” FLAN instruction tuning is in Wei et al. (2022), “Finetuned Language Models are Zero-Shot Learners.” Self-Instruct is in Wang et al. (2022); Alpaca in Taori et al. (2023). Sycophancy in RLHF models is documented in Perez et al. (2022) and Sharma et al. (2023).