# LLM inference explained: where the time and money go

What happens between sending a prompt and reading the answer, what it costs, and how to make it cheaper and faster

Oct 29, 2026 · Inference · 12 min read · https://arclyx.ai/blog/llm-inference-explained

## Summary

LLM inference is what happens when a trained language model answers a prompt. It runs in two phases: **prefill** reads the whole prompt at once, and **decode** writes the answer one token at a time [Source 1]. Prefill decides how long you wait for the first word, and decode decides how long the rest takes. Money is billed by tokens in and tokens out, and on Anthropic's price list every token a model writes costs five times as much as a token it reads [Source 4]. This guide follows one request from start to finish, then shows how to measure your own and which changes make inference cheaper and faster.

## Key Takeaways

- **Two phases, two limits.** Prefill is usually limited by the GPU's compute, decode by how fast it can read memory [Sources 1, 2].
- **Long prompts delay the start; long answers take the time.** Prefill grows with the prompt, decode with every token written [Source 1].
- **Output costs more than input.** In the worked example below, the answer is 12% of the tokens, 40% of the cost and nearly all of the wait.
- **The KV cache makes decode fast and fills the memory.** How a server manages it decides how many requests fit on one GPU [Sources 2, 3].
- **Most savings come before the hardware.** A smaller model, prompt caching, batch pricing, shorter context and capped answers cut the bill without touching a server [Sources 4, 6].
- **Measure before you tune.** Tokens in, tokens out, time to the first token and total time tell you which phase to work on [Source 1].

## What is LLM inference?

LLM inference is running a trained large language model to get an answer. You send text, the model predicts the next token, adds it to the text and repeats until the answer is done. Every chat reply, every API call and every step an AI agent takes is an inference request.

A token is a piece of a word, and it's the unit providers count and bill. How many tokens a text becomes depends on the model's tokenizer: Anthropic says its newer models produce about 30% more tokens for the same text than older ones [Source 4].

### How is inference different from training?

Training builds the model. It runs huge amounts of text through the model and adjusts its weights, a long one-off job on large clusters. Inference uses the finished model: the weights stay fixed, and the work is answering one request after another. Training is the lab's cost of making a model; inference is the running cost of using it, paid on every request.

## How does LLM inference work?

LLM inference works in two phases on top of a tokenizer. The prompt is split into tokens; prefill reads all of them in one pass and stores what it learns in the KV cache; decode then writes the answer one token at a time, reading that cache at every step [Source 1]. Each part has its own limit, which is why they slow down for different reasons.

**How one request runs** (an animated diagram on the page; its steps):

1. **The prompt becomes tokens.** A tokenizer splits your text into tokens, the pieces the model reads and the unit providers bill.
2. **Prefill reads the whole prompt at once.** One pass through the model reads every prompt token together, fills the KV cache and gives the first token of the answer.
3. **Decode writes one token per pass.** Each new token is one more pass. Every pass reads the whole cache and adds one entry to it, so long answers take the time.
4. **One pass to read, one pass per token to write.** Prefill read all 5 prompt tokens in one pass. Decode needed one more pass for each of the other 9 answer tokens.

### The prompt becomes tokens

The model never sees your text directly. A tokenizer splits it into tokens and maps each one to a number. At each step the model takes those numbers and gives a probability for every possible next token; one is picked, turned back into text and streamed to you.

### Prefill: the model reads the whole prompt at once

In prefill the model processes every prompt token in one parallel pass and builds the KV cache, its working memory of the prompt [Source 1]. With all the tokens processed together, there's plenty of arithmetic to keep the GPU busy, so prefill is usually compute-bound [Source 1]. The wait for the first token, called time to first token, is mostly prefill plus time spent in a queue and on the network [Source 1].

Long prompts slow prefill more than you might expect. Redis notes that doubling a long prompt from 16K to 32K tokens roughly quadruples the attention work [Source 1]. That's the attention part, not all of prefill, but it's why long documents and long chat histories add to the wait so quickly.

### Decode: the model writes the answer one token at a time

In decode the model produces one token per step, and each step reads the KV cache [Source 1]. A step does little arithmetic but has to read the model's weights and the whole cache from GPU memory, so decode is limited by memory bandwidth, not compute [Sources 1, 2]. The gap between streamed tokens is the inter-token latency, and it adds up: in an example from Redis, a 500-token answer at 80 ms per token spends about 40 seconds in decode alone [Source 1].

When one server runs both phases on the same GPUs, they get in each other's way. Our deep dive on [prefill-decode interference](https://arclyx.ai/blog/solving-llm-serving-latency-interference) covers how big serving systems keep them apart.

### What is the KV cache, and why does it matter?

At every layer, each token the model has read or written produces a key and a value that later tokens look back at. The KV cache keeps them, so the model doesn't recompute them for the whole text at every step [Source 2]. That's what lets decode add one token per step without starting over.

The cache takes memory, and it grows with every token of every request a server is handling: NVIDIA notes it grows "linearly with batch size and sequence length" [Source 2]. So memory, more than compute, often decides how many requests a GPU can serve at once. The 2023 PagedAttention paper behind vLLM showed how much badly managed cache memory wastes. Storing the cache in pages, the way an operating system stores memory, gave 2–4x the throughput of the serving systems of the time at the same latency [Source 3].

## A worked example: one request, start to finish

Take one request to Claude Sonnet 5.5: a 3,000-token prompt (instructions, a document and a question) and a 400-token answer. The prices are Anthropic's list prices on 29 October 2026, $2 per million input tokens and $10 per million output tokens [Source 4]. The timings are made up to show the arithmetic; they aren't measurements.

**Where the time goes.** Say the first token arrives after 0.5 seconds (prefill, plus any queue and network time) and the rest stream at 20 ms per token. The remaining 399 tokens take about 8 seconds, so the answer is complete after about 8.5 seconds. Waiting for the first token is about 6% of the wait, decode about 94%.

**Where the money goes.**

| | Tokens | Price per million | Cost |
| --- | --- | --- | --- |
| Input (the prompt) | 3,000 | $2 | $0.0060 |
| Output (the answer) | 400 | $10 | $0.0040 |
| **Total** | **3,400** | | **$0.0100** |

That's a cent a request, or $100 for 10,000 of them. The answer is 12% of the tokens but 40% of the cost and nearly all of the wait.

![Three bars for one request to Claude Sonnet 5.5, split into the prompt side and the answer side. Tokens: 3,000 in (88%) and 400 out (12%). Cost: $0.0060 for input (60%) and $0.0040 for output (40%). Time: 0.5 seconds to the first token (6%) and 8 seconds of decode (94%).](https://arclyx.ai/blog/llm-inference-explained/img/time-and-cost-of-one-request.4f410257-1600.webp)

Now change one thing at a time:

- **Cache the repeated part.** If 2,500 of the prompt tokens are the same instructions and document every time, a cache hit bills them at $0.10 per million instead of $2 [Source 4]. Input falls from $0.0060 to $0.00125, and the request costs about $0.0053, roughly half. The request that writes the cache pays a little more for those tokens ($2.50 per million), and the default cache lasts five minutes [Sources 4, 5].
- **Cap the answer.** A 200-token answer takes about 4 seconds of decode instead of 8 and costs $0.002 less.
- **Don't wait for it.** If nobody is waiting on the answer, Anthropic's Batch API halves both prices to $1 and $5 per million, so the request costs $0.005 [Source 4].
- **Run it ten times.** An agent that takes ten rounds sends its growing context back every round, so the input side grows far faster than the answers. That's why [AI agents cost more to run than chat](https://arclyx.ai/blog/the-inference-cost-trap).

## Why is LLM inference slow?

LLM inference is slow because every answer is written one token at a time, and each token means reading the model's weights and its cache from memory again [Sources 1, 2]. Long prompts add a wait before the first word, long answers add time after it, and long chats add both.

### Why does the first word take a moment?

Before the model writes anything it reads your whole prompt, and the request may wait in a queue first [Source 1]. Long prompts, attached documents and long chat histories all add to that wait.

### Why does the answer stream at a steady pace?

Every token takes one more pass through the model, and every pass reads the weights and the cache from memory [Sources 1, 2]. Memory speed sets the pace, so an answer twice as long takes about twice as long.

### Why do long chats get slower and more expensive?

Every turn sends the whole conversation again. The prompt grows, so each turn waits longer for the first word and pays for more input, unless the earlier part is cached [Sources 1, 5]. The KV cache grows too, so each decode step has more to read.

## How much does LLM inference cost?

On a hosted API you pay per token, with one price for input and a higher one for output. On Anthropic's list, output costs five times input on every current model, from $0.10 and $0.50 per million tokens on Claude Haiku 5.5 (for prompts up to 100,000 tokens) to $10 and $50 on Claude Fable 5.1 [Source 4].

### Why do output tokens cost more than input tokens?

Providers don't publish their reasons, but the two phases explain it. Input tokens are read together in one parallel pass. Output tokens are written one per step, and every step reads the weights and the cache from memory [Sources 1, 2]. A GPU writes far fewer tokens a second than it reads, and the price follows the work.

### Is the price per token the whole bill?

No. The bill is price times tokens, and the surprises are in the token count: chat history sent again on every turn, long system prompts, retries, tool results pasted back into the context, and agents that loop. Count the cost of each finished task, not each request. Our post on [the inference cost trap](https://arclyx.ai/blog/the-inference-cost-trap) follows where an agent's tokens go.

## How do you make LLM inference cheaper and faster?

Start with what you send, then how you pay for it, and only then how it's served.

If you call a hosted API:

- **Use the smallest model that does the job.** Input prices for Anthropic's current models run from $0.10 to $10 per million tokens, a factor of 100 [Source 4].
- **Cache what repeats.** Put fixed instructions and documents at the start of the prompt so they can be cached. Cache hits on Claude models cost a tenth of the input price or less [Sources 4, 5].
- **Batch what nobody is waiting for.** OpenAI and Anthropic both take 50% off batch requests; OpenAI's finish within 24 hours [Sources 4, 6].
- **Send less and cap the answer.** Trim old history and bulky tool results, and set a maximum length where a short answer will do.

If you serve models yourself, the levers move into the server:

- **Memory management and batching**, so more requests share each GPU. That's what paged KV caches are for [Source 3].
- **Quantization**, which stores the weights in fewer bits so there's less to read on every step [Source 2].
- **Speculative decoding.** A small, fast drafter proposes several tokens and the main model checks them in one pass, and the output stays the same as the main model's own [Source 7]. On Google TPU v5p, a Google and UCSD team's diffusion-style drafter, DFlash, made vLLM serving of Llama-3.1-8B-Instruct 2.29x faster end to end, against 1.30x for the EAGLE-3 method [Source 8].
- **Separate machines for prefill and decode**, covered in our [latency deep dive](https://arclyx.ai/blog/solving-llm-serving-latency-interference).
- **Hardware, compared with care.** SemiAnalysis's September 2026 preview put Google's TPUv7 Ironwood ahead of Nvidia's B200 and B300 on performance per dollar at one latency target. It also said the lead applied to that point only, and that across most of the raw-performance curve TPUs don't beat Nvidia GPUs [Source 9]. Hardware results only compare on the same model, setup and latency target.

## How do you measure your own inference?

Record five things for each request:

1. **Input tokens**, and how many of them were cached.
2. **Output tokens.**
3. **Time to first token**: when the first streamed token arrived.
4. **Total time**: when the last one arrived.
5. **Cost**, from the provider's usage fields and price list.

Then work out the gap between tokens: inter-token latency = (total time − time to first token) ÷ (output tokens − 1) [Source 1]. Run the same request at a few prompt lengths with a fixed answer length. If the first token slows down sharply as prompts grow, work on prefill: shorter prompts, or caching the part that repeats. If decode is most of the total, work on output: shorter answers, a faster model, or speculative decoding if you run the server. Then add up the cost of each finished task over a week, because one request rarely tells the whole story.

On paid Arclyx plans, each run's page shows its time, tokens and price, and the input, cached and output tokens of every model call, so you can see an agent's inference cost without extra logging. Our guide to [keeping AI agent costs under control](https://arclyx.ai/blog/ai-agent-cost-control) shows where to look, and the [AI agent observability page](https://arclyx.ai/product/observability) shows what else each run keeps.

## Common questions

Two questions come up once the basics are clear: what hardware inference needs, and what the software that serves a model is called.

### Do you need a GPU for LLM inference?

Not if you use a hosted API, because the provider runs the hardware. To serve a large model yourself, you need GPUs or other accelerators in practice, because decode needs fast memory. Small open models with quantized weights can run on an ordinary computer with tools such as llama.cpp, more slowly.

### What is an inference engine?

It's the software that serves a model: it batches requests, manages the KV cache and runs prefill and decode. vLLM, SGLang and TensorRT-LLM are open-source examples. An inference provider runs the engine and the hardware for you and charges per token.

## Sources

1. Redis — Jim Allen Wallace, *Prefill vs Decode: LLM Inference Phases Explained* (28 April 2026, updated 29 April 2026). Redis sells caching for LLM apps. [https://redis.io/blog/prefill-vs-decode/](https://redis.io/blog/prefill-vs-decode/)
2. NVIDIA — Shashank Verma and Neal Vaidya, *Mastering LLM Techniques: Inference Optimization* (17 November 2023). [https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/)
3. Kwon et al., *Efficient Memory Management for Large Language Model Serving with PagedAttention* (SOSP 2023; arXiv, 12 September 2023). [https://arxiv.org/abs/2309.06180](https://arxiv.org/abs/2309.06180)
4. Anthropic — *Pricing*, "Model pricing" and "Batch processing" (Claude Platform docs, checked 29 October 2026). [https://platform.claude.com/docs/en/about-claude/pricing](https://platform.claude.com/docs/en/about-claude/pricing)
5. Anthropic — *Prompt caching* (Claude Platform docs, checked 29 October 2026). [https://platform.claude.com/docs/en/build-with-claude/prompt-caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
6. OpenAI — *Batch API* (OpenAI API docs, checked 29 October 2026). [https://developers.openai.com/api/docs/guides/batch](https://developers.openai.com/api/docs/guides/batch)
7. NVIDIA — Jamie Li, Chenhan Yu and Hao Guo, *An Introduction to Speculative Decoding for Reducing Latency in AI Inference* (17 September 2025). [https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/](https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/)
8. Google Developers Blog — Weiren Yu, Yarong Mu and Lihao Ran (Google), Zhaoxiang Feng, Yiming Zhao and Hao Zhang (UCSD), *Supercharging LLM inference on Google TPUs: Achieving 3X speedups with diffusion-style speculative decoding* (4 May 2026). [https://developers.googleblog.com/supercharging-llm-inference-on-google-tpus-achieving-3x-speedups-with-diffusion-style-speculative-decoding/](https://developers.googleblog.com/supercharging-llm-inference-on-google-tpus-achieving-3x-speedups-with-diffusion-style-speculative-decoding/)
9. SemiAnalysis — *TPU Inference Externalization Full Steam Ahead* (InferenceX preview, 7 September 2026). [https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam](https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam)