Teaching Claude Why
Anthropic alignment methods
Three alignment method families:
SDF: During pretraining, on the base model, introduce pretraining-like documents of:
- Stories of other AI acting according to the Claude Constitution
- Documents discussing the Constitution
Converting these into chat format somehow is less effective.
SFT (chat-formatted transcripts):
- honeypot examples with injected-and-later-removed prompts to the model to explain its ethical reasoning for its actions. You need this ethical reasoning part or else it’s not as effective.
- Claude providing difficult advice to the user in ethical situations.
RL
They are super vague about how they actually do RL.
Unclear if this is RLHF or RLAIF or RLVR. A bunch of harmlessness RL environments with tool call definitions that don’t do anything, just to keep the environment more like the production environments.