Thomas Kalnik's Blog

Vaswani et al. drop recurrence entirely and build a sequence model from self-attention alone. Every model on this page descends from this architecture.

Generative pretraining on BooksCorpus followed by task fine-tuning. A decoder-only transformer turns out to be a strong general-purpose learner.

Bidirectional masked-language-model pretraining sweeps GLUE and SQuAD. For two years, fine-tuning BERT was the default way to do NLP.

Zero-shot task transfer from web-scale pretraining. OpenAI staged the release over concerns about misuse, which itself became a story.

Intra-layer tensor parallelism lets NVIDIA train an 8.3B GPT-2 across 512 GPUs. The parallelism playbook that every large training run still uses.

Everything is text-to-text. The C4 corpus and a systematic study of transfer learning choices make T5 a workhorse for years.

Kaplan et al. show loss falls as a smooth power law in parameters, data, and compute. This paper turned model building into a budgeting problem.

175B parameters and in-context learning: describe a task in the prompt and the model does it. The API launched in June and seeded the first wave of LLM startups.

Sparse mixture-of-experts routing scales a model to 1.6 trillion parameters while keeping per-token compute fixed. MoE becomes the path to trillion-scale.

A volunteer collective trains a GPT-3-class model on The Pile with TPUs and releases the weights. The open-model movement has its first real base model.

Low-rank adapters freeze the base weights and train small update matrices instead. Fine-tuning a big model on one GPU becomes routine.

OpenAI Codex, a GPT model fine-tuned on public code, ships inside the editor as GitHub Copilot. AI-assisted coding goes from demo to daily tool.

The first GPT-3-scale model from outside OpenAI, with a larger 256K-token vocabulary.

The largest dense model of its time, trained with DeepSpeed and Megatron across 560 DGX A100 nodes.

DeepMind enters the scaling race with a 280B model and a detailed analysis of where scale helps and where it does not.

Supervised fine-tuning plus reinforcement learning from human feedback. Labelers preferred a 1.3B InstructGPT over 175B GPT-3, and this became the ChatGPT recipe.

Asking a model to show its work before answering unlocks multi-step reasoning, but only at scale. The seed of the later reasoning models.

The largest fully open model at release, trained on CoreWeave GPUs. Its training library became a common base for open efforts.

Compute-optimal training wants roughly 20 tokens per parameter. A 70B model trained on 1.4T tokens beats 280B Gopher, and every lab retunes its data budgets.

540B dense parameters trained on 6,144 TPU v4 chips with the Pathways system. Breakthrough scores on reasoning and code, and the base for Med-PaLM.

Meta replicates GPT-3 and releases the weights to researchers along with the full training logbook, including every crash and restart.

An IO-aware exact attention kernel that tiles computation to stay in SRAM. Longer context and faster training for free; now standard in every stack.

A 176B multilingual model trained openly by a thousand-researcher collaboration on the Jean Zay supercomputer.

Not a language model, but the open-weights release that proved generative models could be run and fine-tuned by anyone with a consumer GPU.

Instruction tuning on 1,800 tasks. Flan-T5 checkpoints became the go-to small instruction-following models before Llama.

A free chat interface on GPT-3.5 reaches 100 million users in two months, the fastest consumer product adoption ever. The industry reorganizes around it.

Chinchilla-style models from 7B to 65B, released to researchers and leaked within a week. The 7B model running on a laptop kicks off the open fine-tuning boom.

LLaMA-7B instruction-tuned on 52K GPT-generated examples for under $600. Proof that a usable assistant could be distilled cheaply.

Multimodal input, top-decile bar exam performance, and a technical report that disclosed nothing about size or architecture. Widely reported as a 1.8T MoE.

Anthropic's first public models, trained with Constitutional AI: RL from AI feedback against a written set of principles instead of only human labels.

Google's answer to GPT-4 powers Bard and Workspace. Smaller than PaLM but trained on far more tokens, following Chinchilla.

The first 100K-token context window from a frontier lab: a whole novel in one prompt when everyone else offered 4K to 32K.

4-bit NormalFloat quantization plus LoRA fine-tunes a 65B model on a single 48GB GPU with no loss in quality. The technique behind most hobbyist fine-tunes.

Preference tuning without a reward model or RL loop: a simple classification loss on chosen versus rejected pairs. It replaced PPO in most open post-training pipelines.

Claude gets a public chat interface at claude.ai, stronger coding and math, and the 100K window as standard.

Open weights with a commercial license, plus chat models tuned with RLHF. Llama 2 became the base for thousands of products and fine-tunes.

Llama 2 continued-pretrained on code with infilling and 100K-token extrapolation. A 70B version followed in January 2024.

Abu Dhabi's Technology Innovation Institute releases the largest open model to date, trained on 3.5T tokens of RefinedWeb.

ChatGPT can now see, hear, and speak: image input, voice conversation, and DALL·E 3 image generation arrive in the same week.

Released as a magnet link under Apache 2.0. Grouped-query and sliding-window attention let a 7B model beat Llama 2 13B, and Europe gets a frontier lab.

xAI's first model, a 314B mixture of experts, built in four months. The weights were open-sourced under Apache 2.0 in March 2024.

DevDay brings a 128K context window, JSON mode, cheaper pricing, the Assistants API, and custom GPTs.

200K context, halved hallucination rates, system prompts, and beta tool use.

A quant fund's research arm releases its first open models, along with scaling-law experiments on hyperparameters. Few noticed at the time.

Alibaba's Qwen series steps up to 72B with open weights and 32K context, starting a release cadence that would make Qwen the most downloaded open family.

Selective state-space models match transformers at small scale with linear-time sequence processing. The strongest challenger to attention so far, later hybridized into production models.

Google DeepMind's first natively multimodal family in Ultra, Pro, and Nano sizes. Ultra was the first model to claim a win over GPT-4 on MMLU.

The first strong open mixture-of-experts model matches GPT-3.5 while running as fast as a 13B dense model. Sparse MoE becomes the open-model default.

Textbook-quality synthetic data lets a 2.7B model rival models ten times larger. Data quality becomes a first-class scaling lever.

A one-million-token context window with near-perfect recall, later extended to two million. An hour of video or a whole codebase fits in a single prompt.

Three tiers with native vision. Opus was the first model to clearly beat GPT-4 on standard benchmarks, ending a year of OpenAI's uncontested lead.

A larger open MoE with 64K context and native function calling, again shipped first as a torrent.

Trained on 15 trillion tokens, far past Chinchilla-optimal, because inference cost matters more than training cost. The 8B model became the default small open model.

Multi-head latent attention compresses the KV cache by 93 percent, and fine-grained MoE with shared experts cuts training cost. Priced so low it triggered a price war in China.

One model trained end-to-end on text, audio, and vision, responding to speech in about 300 milliseconds. Free-tier ChatGPT users get GPT-4-class intelligence.

Five sizes from 0.5B to 72B under Apache 2.0, with 128K context and strong multilingual coverage.

A mid-tier model that beat Opus at twice the speed, with Artifacts for live code and documents. It became the default model for serious coding work.

Google's open family gets a 27B model trained with knowledge distillation that punches above its size on the Chatbot Arena.

The first open-weights model at GPT-4 level, trained on 16,000 H100s, with a 92-page report on data, infrastructure, and failure modes. 128K context across the family.

A 123B dense model with open weights for research, released the day after Llama 3.1 to stay in the frame.

xAI reaches GPT-4-class performance and adds image generation via Flux inside X.

The first reasoning model: reinforcement learning teaches the model to think in a hidden chain of thought before answering. Test-time compute becomes a second scaling axis.

Seven sizes trained on 18T tokens, plus Coder and Math variants. Qwen2.5-72B matched Llama 3.1 405B on many benchmarks and became the favorite base for fine-tuning.

Vision models at 11B and 90B, and 1B and 3B text models built for phones and edge devices.

An upgraded Sonnet and the first public beta of computer use: the model moves a cursor, clicks, and types to operate real software.

Post-training improvements bring the 70B model up to 405B-level quality at a fraction of the cost.

Google calls it the start of the agentic era: native tool use, multimodal output, and Deep Research, with a Flash model outperforming the previous Pro.

o3 scores 87.5 percent on ARC-AGI at high compute and 25 percent on FrontierMath, benchmarks built to resist LLMs. The full o1 had shipped two weeks earlier.

A 671B MoE trained in FP8 on 2,048 H800s for a reported $5.6M of compute, matching GPT-4o and Claude 3.5. MLA, auxiliary-loss-free load balancing, and multi-token prediction.

An o1-class reasoning model under MIT license, with a paper showing reasoning emerging from pure RL with GRPO. A week later it wiped $600B off Nvidia in a day.

A fast, cheap reasoning model with selectable reasoning effort, shipped to free users days after R1.

Trained on the 200,000-GPU Colossus cluster with ten times Grok 2's compute. Adds Think mode and briefly tops the Chatbot Arena.

The first hybrid reasoning model: instant answers or visible extended thinking from the same weights. Claude Code launches as a terminal agent in research preview.

OpenAI's largest pretraining run, with better taste and fewer hallucinations but no reasoning. Expensive, and retired from the API within months.

Open multimodal models up to 27B with 128K context, built to run on a single GPU or TPU.

Google's first thinking model debuts at the top of every leaderboard with a million-token window. The moment Google was back at the frontier.

Meta's first MoE models, natively multimodal, with Scout claiming a 10M-token context. A benchmark-tuned Arena variant sparked controversy, and Behemoth never shipped.

API-only models with a million-token context and better instruction following, aimed at developers building agents.

Reasoning models that call tools mid-thought: web search, Python, and image manipulation inside the chain of reasoning. Codex CLI ships alongside.

Eight open models from 0.6B to a 235B MoE with a thinking mode you can toggle per request, trained on 36T tokens across 119 languages.

Built for long-running agentic work: Opus 4 ran autonomously for hours on refactoring tasks. Claude Code goes GA and coding agents become the center of the market.

RL compute scaled to match pretraining. Grok 4 Heavy runs parallel agents and posts the top Humanity's Last Exam score of the summer.

A trillion-parameter open MoE trained stably with the Muon optimizer, tuned for agentic tool use. The largest open model yet.

A 480B open coding MoE with 256K native context and its own agent CLI, closing in on Claude Sonnet 4 on agentic coding.

OpenAI's first open-weight models since GPT-2, under Apache 2.0. The 120B MoE runs on a single 80GB GPU with o4-mini-level reasoning.

An incremental Opus with 74.5 percent on SWE-bench Verified and better multi-file refactoring.

A unified system that routes between a fast model and a deeper reasoning model. Strong on coding and health, with a launch overshadowed by users mourning GPT-4o.

Thinking and non-thinking modes in one checkpoint, better tool use, and a UE8M0 FP8 format chosen with domestic Chinese accelerators in mind.

Anthropic's strongest coding model at the time, capable of 30-hour autonomous sessions, with a memory tool and context editing for long-running agents.

Sonnet 4 coding performance at a third of the cost, with extended thinking and computer use in the smallest tier.

An open reasoning agent that chains 200 to 300 tool calls, briefly leading closed models on Humanity's Last Exam and BrowseComp.

Warmer default personality, adaptive reasoning that skips thinking on easy prompts, and a Codex-Max variant that compacts context across sessions.

A quieter two-week rollout focused on emotional intelligence, lower hallucination, and a Fast variant for tool calling.

Shipped straight into Search, the Gemini app, and Vertex on day one, with record LMArena and ARC-AGI-2 scores and a Deep Think mode. Google's strongest lead to date.

Opus pricing cut by two thirds, an effort parameter to trade cost for capability, and automatic context compaction for long agent runs.

DeepSeek Sparse Attention cuts long-context cost by half, and a Speciale variant reaches gold-medal level on IMO and IOI problems.

A rapid response to Gemini 3: better knowledge work, spreadsheets, and long-context reasoning, followed by a Codex variant in January.

Pro-level reasoning at Flash speed and price, and the new default model in the Gemini app.

Native vision and an agent swarm mode that fans work out to parallel sub-agents, keeping open models within reach of the closed frontier.

A million-token context by default, 128K output, adaptive thinking, and agent teams in Claude Code that coordinate multiple Claude instances on one task.

A coding-specialized model released the same day as Opus 4.6, as the two labs trade blows on agentic software engineering.

Zhipu's open flagship joins DeepSeek, Qwen, and Kimi in a Chinese open-weights quartet that now sets the pace for open models.

The mid tier gets the million-token window and a large jump in computer-use reliability.

77.1 percent on ARC-AGI-2, more than double Gemini 3 Pro, and 80.6 percent on SWE-bench Verified, aimed at long-horizon agentic workflows.

Built-in computer use with 75 percent on OSWorld-Verified, up from 47 percent, and a third fewer factual errors than GPT-5.2. Mini and nano variants followed on March 17.

Reasoning and non-reasoning variants plus a multi-agent beta, with the emphasis on speed and low hallucination.

A 31B dense model and a 26B MoE with only 4B active parameters, keeping Google's open line competitive with the Chinese labs.

Anthropic reveals a model too capable at offensive cybersecurity to release, and instead gives 40-plus partners access through Project Glasswing to harden critical software.

Meta Superintelligence Labs' first model, closed-weights and natively multimodal, replaces Llama as the engine of Meta AI. The end of Meta's open-weights era, at least for its flagship.

Higher-resolution vision, an xhigh effort level, task budgets, and file-system memory recall for agents.

82.7 percent on Terminal-Bench 2.0 and 35 percent on FrontierMath Tier 4, with a 71 percent pass rate on expert cybersecurity tasks in AISI testing.

Trillion-scale open MoE under MIT license with a native million-token window and 384K output, near parity with GPT-5.4. Flash weights followed in July; Pro went GA on August 13.

Max, Plus, Flash, and a 35B MoE with 3B active. Qwen's point-release cadence is now roughly monthly.

Announced at I/O with a promise of 3.5 Pro the following month. The Pro model still has not shipped.

Focused on code honesty: four times less likely to miss flaws in its own work, plus dynamic workflows in Claude Code.

A new Mythos-class tier above Opus, always-on adaptive thinking, and state of the art across the board. Suspended June 12 under a US export-control directive, restored June 30 with new safeguards. Fable later helped disprove the 87-year-old Jacobian conjecture.

Near-Opus intelligence at $2 per million input tokens, with adaptive thinking and xhigh effort. The new default for free and Pro plans.

Built on xAI's 1.5-trillion-parameter V9 foundation and co-developed with Cursor for coding. Grok 4.6 followed in August; Grok 5 is still training.

Three tiers named for the sun, earth, and moon. Sol tops the Artificial Analysis coding index while using half the tokens of Fable 5, and OpenAI calls it its strongest cybersecurity model.

Moonshot's third-generation open model, with a Fast variant, keeps the open frontier within a few months of closed models.

Three Flash-tier releases in one day, including a cyber-specialized model, while the 3.5 Pro flagship slips again. 3.7 Flash and 3.8 Flash followed in August and September.

A step change at the Opus tier, close to Fable 5 at half the price, with thinking on by default, a full effort ladder, and a fast mode at 2.5 times the speed.

Alibaba's flagship open model, followed by 27B and Flash variants within the month.

A 30B open model under Apache 2.0 that runs offline on a 24GB consumer GPU: Meta keeps a foot in open weights below its closed flagship.

Cache reads cut to a quarter of the price, per-message effort changes mid-conversation, and content provenance. Mythos 5.1 stays limited to Glasswing partners.

The fourth Flash point release since July, as Google waits on 3.5 Pro and starts the Gemini 4 pretraining run.

OpenAI's generational leap, held back four weeks because it crossed the Critical cyber threshold in the Preparedness Framework. Saturates FrontierMath Tier 4 at 98 percent, ARC-AGI-3 at 99.9 percent, and ExploitBench at 100 percent. Brockman says it may come to be seen as AGI.