Part 2 of 10 in The Price of a Thought, a weekly series on how cheap tokens shaped the AI landscape of 2026. Last week: why cheaper AI means more AI.
The unit
Language models do not read or write words. They work in tokens, which are chunks of text averaging about three-quarters of a word. Everything is priced per million tokens, with one rate for what you send in and a higher rate for what the model writes back.
What happens inside, per token
Inside, a model is a tall stack of very large tables of numbers, called weights or parameters. To produce one token, the model takes a numerical representation of the text so far and pushes it through every layer of the stack. Each layer multiplies it by that layer's tables. Each weight gets used once, in one multiply and one add. So the arithmetic needed for a token is about two operations for every weight the token passes through.
That gives a simple rule: compute per token is roughly proportional to the number of active parameters. A model with 500 billion active parameters performs about a trillion operations to emit one token. A model with 20 billion performs about 40 billion. The same chip produces around 25 times as many tokens per second with the smaller model, so each token costs around a twenty-fifth as much.
Three details that explain every price list
Input is cheaper than output. A model can process all the tokens of a prompt in parallel, in one sweep. But it writes its answer one token at a time, each requiring a fresh pass through the stack. That is why output is usually priced at several times input: recent flagship lists show $5 in and $25 out for one model, $10 in and $50 out for another, a five-to-one ratio.
Context costs money. Each new token must attend to everything before it, so a long conversation or a huge document makes every subsequent token more expensive to produce. Providers reduce this with caching: if the start of your prompt is identical to a recent one, they reuse the earlier work. On the same price list, cached input is offered at a tenth of the normal input rate.
The true bottleneck is often memory, not arithmetic. For each token, the chip has to fetch every active weight from memory. The speed at which it can move those bytes limits how fast tokens come out. This is why AI chips are judged heavily on memory bandwidth, and why the newest ones advertise it prominently.
The equation
Put together, the cost of a token is three things multiplied: how many operations the token needs, what each operation costs on the hardware, and how fully the hardware is kept busy.
If your team is paying an AI bill today, those three terms are also the only three places the bill can come down. Shorter prompts and cache-friendly prompt structure attack the first. Picking the right hardware or host attacks the second. Batching work that can wait attacks the third.
Next week: the six ways the industry has pushed the price of a token down, and how far it has fallen.