Recursively Summarizing Books with Human Feedback
OpenAI's "Recursively Summarizing Books with Human Feedback" is the nearest real paper whose title contains both "recursive" and "human feedback", and the user-facing phrase is better read as a slip…
- Date
- 22 September 2021
- Who
- OpenAI
- People
- Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano
- Confidence
- High
- Deep dive
- RLHF and instruction tuning (how base models became assistants)
Tier: Supporting · Significance: 3/5 · Org(s): OpenAI · People: Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano · Confidence: High OpenAI's "Recursively Summarizing Books with Human Feedback" is the nearest real paper whose title contains both "recursive" and "human feedback", and the user-facing phrase is better read as a slip for RLHF (see the Terminology box). A GPT-3 model (175B and 6B variants) first summarizes small sections of a book, then summarizes those summaries, and so on up a tree; humans give feedback (demonstrations and comparisons) only on small tasks, using lower-level summaries to help judge higher-level ones, so that labelers could supervise models without having read the whole books. Whole-book summaries matched human-written quality in about 5% of cases for the best 175B model, with the best BookSum results published at the time (arXiv:2109.10862, v1 2021-09-22). Its purpose was an empirical test of the scalable-oversight idea in B05-04, and InstructGPT's discussion cites it as an example of a task that is hard for humans to evaluate directly (InstructGPT §5.1). Sources: Wu et al.