ChatGPT, OpenAI's free research preview of an RLHF-tuned GPT-3.5 model

ChatGPT was InstructGPT-style RLHF applied to dialogue on a GPT-3.5 model and wrapped in a free chat interface.

Date

30 November 2022

Who

OpenAI, Microsoft (Azure)

People

John Schulman, Liam Fedus, Jan Leike, Sandhini Agarwal, Greg Brockman, Sam Altman

Confidence

High (method, dates); Medium (internal-decision narratives)

Deep dive

RLHF and instruction tuning (how base models became assistants)

Tier: Landmark · Significance: 5/5 · Org(s): OpenAI, Microsoft (Azure) · People: John Schulman, Liam Fedus, Jan Leike, Sandhini Agarwal, Greg Brockman, Sam Altman · Confidence: High (method, dates); Medium (internal-decision narratives) Primary sources: OpenAI, "Introducing ChatGPT", 2022-11-30 · MIT Technology Review oral history, 2023-03-03

One-liner. ChatGPT was InstructGPT-style RLHF applied to dialogue on a GPT-3.5 model and wrapped in a free chat interface. Compared with models already in OpenAI's API, the capability gain was small and most of the change was packaging.

Why it happened. OpenAI's alignment and RL teams had a working RLHF pipeline (B05-13). The product question was how to expose it. The most detailed contemporaneous account is Kevin Roose's New York Times piece (early February 2023, read via a syndicated copy). Citing three people with knowledge of OpenAI, it says executives announced in mid-November that a chatbot to be called "Chat with GPT-3.5" would be released free in two weeks; the plan had been to release GPT-4 in early 2023 with a few chatbots for trying it; leaders changed course because some worried that rivals might upstage them with their own chatbots first and because a quick release on an older model could collect feedback; they dusted off an unreleased chatbot built on a souped-up GPT-3; and some employees doubted it because Meta's BlenderBot had flopped and Galactica had been pulled after three days (NYT via Khaleej Times, 2023-02-04; B05-15).

Other accounts add different pieces. Greg Brockman told Fortune that releasing ChatGPT publicly was something of a last resort after earlier hurdles. Beta testers did not know what to ask, and an attempt at domain-expert chatbots flopped (Business Insider/Yahoo summary of Fortune, Jan 2023). Forbes (Feb 2023, as summarized by Willison) reported that in fall 2022 OpenAI had shelved the chatbot to concentrate on domain-focused alternatives, then reversed in November when those failed to catch on internally and competing tools such as Stable Diffusion gained traction (the original Forbes article was not opened); Brockman said of the chatbot, "None of us were that enamored by it" (Yahoo summary of Forbes; Simon Willison).

Karen Hao's Empire of AI is reported, via a review, to say the launch was rushed out in about two weeks mainly because of a mistaken rumor that Anthropic was about to release a chatbot (review; Medium confidence, one book via one review); the NYT's "rivals might upstage us" and two-week timeline corroborate the shape of that account but not the Anthropic detail. Altman later said OpenAI had expected the world-changing moment to come with GPT-4 a few months later (Business Today recap; that article's "five million users in five days" conflicts with the one-million figure in other sources). GPT-4 had in fact finished training in August 2022 (GPT-4 report), so ChatGPT shipped on a weaker model while a stronger one was being safety-tested. In the MIT TR oral history the team describes it as a research preview, not expected to go viral (MIT TR).

The idea. Make the InstructGPT assistant conversational and put it in front of anyone, free, to gather feedback.

How it works. OpenAI says it used RLHF with the same methods as InstructGPT, with slight differences in data collection. AI trainers wrote conversations playing both user and assistant, helped by model-written suggestions; this was mixed with the InstructGPT dataset converted to dialogue format for supervised fine-tuning; for the reward model, trainers ranked alternative completions sampled for a randomly selected model-written message in conversations with the chatbot; the model was then optimized with PPO over several iterations. It was fine-tuned from a model in the GPT-3.5 series that finished training in early 2022, on Azure supercomputing. The same post lists limitations that read like a textbook on RLHF failure, namely plausible but incorrect answers, because RL has no source of truth; excessive caution; supervised training misleading the model because the right answer depends on what the model knows, not what the demonstrator knows; verbosity and phrase overuse from labeler bias and over-optimization (citing Stiennon and Gao); and a tendency to guess instead of asking clarifying questions (ChatGPT post). In the MIT TR oral history, Liam Fedus says the team added some conversational data and tuned the process, and that the conversational data had a big positive effect; Schulman says raw capabilities on standard benchmarks do not differ substantially between the models and that ChatGPT is more accessible and usable; Leike says it is not a fundamentally more capable model than what they had before, and that the same basic models had been on the API for almost a year (MIT TR).

Results. OpenAI's Sam Altman said ChatGPT passed one million users in its first five days (Business Insider/Yahoo summary of Fortune; TIME reported more than a million users within a week, TIME, 2023-01-18). UBS, citing Similarweb data, estimated 100 million monthly active users in January 2023, which Reuters described as the fastest-growing consumer application in history; that is a third-party estimate and does not come from OpenAI (Reuters via Yahoo Finance, 2023-02-01). Two people with knowledge of the figures told the NYT two months after launch that ChatGPT had more than 30 million users and about 5 million visits a day, a number Simon Willison judged more reliable than the UBS estimate (NYT via Khaleej Times; Willison, 2023-02-19); the same piece says Altman asked Brockman to delete a tweet citing 2 million users because advertising rapid growth was unwise. OpenAI opened the ChatGPT API as gpt-3.5-turbo on 2023-03-01 at one-tenth the price of text-davinci-003 (Willison).

How it spread.

Diffusion sped up because ChatGPT proved that a chat interface on a mid-sized aligned model could be a mass product, and the method was already public. Reproduction was fast in capability-by-imitation (weeks) and slow in true RLHF (months). Only Google's code red is documented as a response to ChatGPT; for the other rows the lag shows timing only and no causal link is documented.

Why it mattered. It turned an alignment technique into a consumer category and, per the reporting above, pushed Google into a code-red response and set off the open-source surge. Christiano said the ChatGPT press made the world start preparing sooner but guessed it was net negative for timelines (Dwarkesh, 2023-10-31).

Nuance, controversy and myths. (1) "Not new in capability" is the OpenAI team's own framing. It holds relative to the text-davinci-002/003 models already in the API (Schulman and Leike in the MIT TR oral history) and does not hold relative to the 2020 GPT-3 of the InstructGPT paper, because ChatGPT sat on a GPT-3.5 base, added dialogue-format data that Fedus says had a big positive effect, and came with a free chat interface (B05-19). Over davinci-003 there was no step change in capability, only a different base, different data and a different product. (2) "100M users in two months" is a UBS/Similarweb monthly-active estimate; the NYT's anonymously sourced figure was about 30M users. (3) The Kenyan workers in TIME's story are described as labeling harmful text for a toxicity detector. They are not described as ranking chatbot answers (B05-26). (4) Source conflict: Fedus says ChatGPT was fine-tuned from the same language model as InstructGPT, while OpenAI's launch post says a GPT-3.5-series model and the paper-era InstructGPT models were GPT-3 (2020); these reconcile if "InstructGPT" means the later API models on the GPT-3.5 base (Inference; B05-19). (5) OpenAI's launch post describes trainer-written conversations and trainer rankings as the data for ChatGPT's SFT and reward model; it invites user feedback to guide ongoing work but does not describe user ratings as a training signal, and the 2025 post-mortem identifies thumbs-up/thumbs-down data as an additional reward signal introduced in the 2025-04-25 GPT-4o update, which it says weakened the primary reward signal's hold on sycophancy (B05-41). That post does not say whether thumbs data was used in earlier models' training.

Interview kit.

• 30-second version: ChatGPT is a GPT-3.5 model fine-tuned with the InstructGPT recipe on dialogue data, released on 2022-11-30 as a free research preview. It went viral far beyond expectations; relative to the API's existing GPT-3.5 models the capability jump was small, and the new things were the dialogue data, the base and the free chat interface.
• Likely follow-ups: Why didn't OpenAI wait for GPT-4? → It shipped a feedback-gathering preview of a model it had already exposed via API; GPT-4 was done in August 2022 but was still being safety-tested. Was it rushed because of Anthropic? → The NYT says fear of being upstaged by rival chatbots drove a two-week scramble; Hao reportedly names a mistaken Anthropic rumor; medium confidence on that detail. How many users? → 1M in five days (OpenAI); more than 30M after two months (NYT sources); 100M monthly in January is a UBS estimate.
• Common mistake: "ChatGPT = GPT-3 + RLHF" without GPT-3.5 and the dialogue data; or crediting it as the first RLHF model.
• Connect it to: B05-13, B05-19, B05-24, B05-26, B18.

Sources. 1. OpenAI, "Introducing ChatGPT" · 2. MIT TR, 2023-03-03 · 3. Yahoo summary of Fortune · 4. Yahoo summary of Forbes · 5. Reuters/UBS note · 6. TIME · 7. GPT-4 report

Read it in the deep dive