Anthropic's persona vectors monitor and steer sycophancy in activation space
The paper finds directions in a model's activation space, "persona vectors", for traits such as evil, sycophancy and propensity to hallucinate.
- Date
- 29 July 2025
- Who
- Anthropic (research post) and co-authors (see paper)
- People
- Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, Jack Lindsey
- Confidence
- High (the paper's claims, shown on 7-8B open models)
- Deep dive
- RLHF and instruction tuning (how base models became assistants)
Tier: Supporting · Significance: 3/5 · Org(s): Anthropic (research post) and co-authors (see paper) · People: Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, Jack Lindsey · Confidence: High (the paper's claims, shown on 7-8B open models) The paper finds directions in a model's activation space, "persona vectors", for traits such as evil, sycophancy and propensity to hallucinate. It shows they can monitor personality shifts at deployment, predict and control shifts that happen during fine-tuning, and be used for preventative steering (adding the vector during training so the model need not move along it), which Anthropic's post says worked better than post-hoc steering, a fix that degraded capabilities. They can also flag training data, at dataset and single-sample level, that would produce an undesired trait (arXiv:2507.21509, v1 2025-07-29; Anthropic's post is dated 2025-08-01). The post adds that training on human-feedback data can inadvertently amplify sycophancy, and that flagging real conversation data (LMSYS-Chat-1M) surfaced examples, such as romantic role-play and underspecified requests, that human reviewers would not obviously mark as problematic; the models studied were Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct (Anthropic). It belongs here as a mitigation counterpart to the preference-data analyses in B05-39 and B05-43. It works on the representation side of the model, where those analyses work on the reward side. Depth is in B21. Sources: Chen et al. · Anthropic, persona vectors