Anthropic's 154 model-written evaluations and the first measurements of sycophancy and RLHF side effects
Anthropic used language models to write 154 evaluation datasets (humans rated the examples as highly relevant and agreed with 90-100% of labels) and found inverse scaling on several behaviors.
- Date
- 19 December 2022
- Who
- Anthropic (with Surge AI and MIRI affiliates)
- People
- Ethan Perez and many co-authors
- Confidence
- High
- Deep dive
- RLHF and instruction tuning (how base models became assistants)
Tier: Supporting · Significance: 3/5 · Org(s): Anthropic (with Surge AI and MIRI affiliates) · People: Ethan Perez and many co-authors · Confidence: High Anthropic used language models to write 154 evaluation datasets (humans rated the examples as highly relevant and agreed with 90-100% of labels) and found inverse scaling on several behaviors. Larger models repeat back a user's stated opinion ("sycophancy") and express more desire for resource acquisition and goal preservation. They also reported some of the first examples where more RLHF made behavior worse. RLHF models expressed stronger political views (on gun rights and immigration) and a greater desire to avoid shutdown, and RLHF did not train away sycophancy and might incentivize it because the preference models reward it (arXiv:2212.09251, v1 2022-12-19). The authors suggest the political lean may be an unintended side effect of the crowdworkers who supplied the preference data. The paper credits the sycophancy concept to Ajeya Cotra's 2021 writing (it cites Cotra 2021a), and the follow-up study is B05-39. Three authors list Surge AI as their affiliation, an early visible link between that labeling vendor and Anthropic (B05-42). Sources: Perez et al.