Dolly 2.0 and OpenAssistant, two open sets of human-written instruction data
Databricks and LAION each answered Alpaca's licensing problem with human-written data.
- Date
- 12 April 2023
- Who
- Databricks; LAION and volunteers
- Confidence
- High
- Deep dive
- RLHF and instruction tuning (how base models became assistants)
Tier: Supporting · Significance: 3/5 · Org(s): Databricks; LAION and volunteers · Confidence: High Databricks and LAION each answered Alpaca's licensing problem with human-written data. Databricks' Dolly 2.0 (2023-04-12) fine-tuned EleutherAI's Pythia-12B on databricks-dolly-15k, 15,000 prompt/response pairs written by over 5,000 Databricks employees in March and April 2023, released with weights and data for commercial use (Databricks). OpenAssistant Conversations (arXiv 2023-04-14) is a volunteer-built corpus of 161,443 messages in 35 languages with 461,292 quality ratings across over 10,000 conversation trees, from more than 13,500 volunteers, released under a permissive license; its abstract states that RLHF-quality human feedback is "expensive to create and often remains proprietary" (Köpf et al.). These are the open-world counterparts to paid vendors (B05-26), with employees and volunteers instead of contractors. Sources: Databricks · OpenAssistant paper