Part 3 of 10 in The Price of a Thought. Last week: what a token is, and the three terms that set its cost.
The first GPT-3 API, in June 2020, was listed at $60 per million tokens. By early 2026, economy models of equal or better quality were on sale for about ten cents: a 600-fold fall in under six years. Andreessen Horowitz's analysis put the trend at tenfold per year for a model of equivalent performance, faster than PC compute or dotcom-era bandwidth. Epoch AI, measuring by benchmark, found yearly declines ranging from 9x to 900x depending on the task.
Six levers produced that curve. Five reduce cost. The sixth turns lower cost into lower price.
1. Fewer operations for the same intelligence
This is the largest effect, and it comes from better training, not better chips. The main method is distillation. A large teacher model generates answers and worked reasoning, and a small student model is trained to imitate them. The student learns from clean, consistent examples of good thinking instead of the noisy sprawl of the internet, so it absorbs far more capability per parameter than a model of its size could a few years ago. The lab pays the teacher's cost once, during training. Every customer query afterward runs on the cheap student. The mini, flash, luna and haiku tiers from the big labs are largely products of this process.
Alongside distillation, the data improved (curated, deduplicated, heavily synthetic), the training recipes improved (small models used to be undertrained), and models learned to use tools. A model that can search the web and run code does not need to memorize every fact in its weights. It needs to reason and to know how to look things up, which takes fewer parameters.
2. Using only part of the model
A mixture-of-experts design holds many specialist sub-networks and routes each token through only a few of them. You store a huge model but pay for a small one on each token. One widely studied open model activates 37 billion of its 671 billion parameters at inference, an 18-fold cut in compute per token. Most large models now use some version of this idea.
3. Cheaper operations
Weights were originally stored as 16-bit or 32-bit numbers. Quantization stores them in 8 or 4 bits. The count of operations does not change, but each is cheaper, and far fewer bytes have to move through memory. Done well, the loss in quality is small. This is also what lets a multi-billion-parameter model fit in a laptop's memory.
4. Better hardware
Each generation of AI chips delivers more tokens per dollar and per watt. The current transition is from Nvidia's Blackwell generation to its successor, Vera Rubin. Nvidia's launch pitch promised up to five times the inference performance and as much as a tenfold cut in cost per token against Blackwell, and the first rack at a major GPU cloud was powered on in June 2026.
Treat the headline with care: those figures are the vendor's own, not independent benchmarks, and silicon-level gains do not pass one-for-one into what end users pay. Still, the direction is clear, and it is telling that Nvidia chose to advertise cost per token instead of raw speed. That is now the number the industry competes on.
Nvidia is also no longer the only option. Google runs its own chips. Amazon has its own. Broadcom designs custom AI chips for Google, Meta, Anthropic and OpenAI. Chips built for one job, serving a known model at high volume, can beat general-purpose GPUs on cost.
5. Better serving software
A chip serving one user at a time wastes most of its capacity. Serving systems batch many users' requests together so they share the cost of reading the weights. They cache repeated prompts. They use small models to draft tokens that the big model then verifies in bulk. And large providers keep their fleets busy around the clock by offering discounts for work that can wait: one lab's batch service takes a flat 50% off both input and output. All of this raises the utilization term.
6. Competition
The five levers above reduce cost. Competition is what turns lower cost into lower price. The key force here is open-weight models: models whose weights are published, so any company with chips can serve them. When a strong open model exists, dozens of hosts compete to serve it, and its price falls toward the cost of electricity and hardware. That puts a ceiling on what closed labs can charge for comparable capability.
Next week: the twist. There are two price curves in AI, not one, and the second isn't falling at all.