The Token Generation Speed Simulator estimates how long one language-model response takes, then separates the wait before the first visible token from the time spent producing the remaining output. It is useful when a chat interface feels slow but the underlying cause is unclear: a long prompt, a slow decoder, or fixed service overhead. Put measured or proposed throughput figures beside the prompt and response sizes you expect to serve.
Loading calculator...
The calculator reports seconds with two decimal places. Its figures are a throughput estimate, so they give a planning baseline for a particular request shape rather than a service-level guarantee across queues, retries, streaming behavior, and concurrent users.
How to use it
Start with Prompt tokens. This is every token processed before generation begins: system instructions, chat history, retrieved passages, tool definitions, and the current user message. Use a percentile from request logs if the prompt size varies. A single average can hide a large tail of long retrieval requests.
Set Output tokens to the response length you allow or usually observe. The simulator treats this as generated tokens, with each token taking time at the selected decode rate. A 500-token answer and a 2,000-token answer can share the same first-token wait while their total times differ sharply.
Check measured throughput on the same model, quantization, batch size, and serving stack before using a value in a launch estimate.
Enter Prefill speed in tokens per second. Prefill processes the prompt before output starts. Then enter Decode speed, also in tokens per second, for sequential output generation. The interface begins with 4,000 tok/s for prefill and 60 tok/s for decode, illustrating that the two stages run at very different rates.
Finally add Fixed overhead in milliseconds. This is converted to seconds and added only to time to first token. It can represent request routing, serialization, an API gateway, or a deliberate buffering delay. Read the four result tiles together, then change one field at a time to see which phase moves.
Fields explained
Prompt tokens – the input processed before the first generated token. The default is 2,000 tokens and the control accepts whole tokens with a minimum of 0.
Output tokens – the number of tokens to generate after prefill. The default is 500 tokens, entered in whole tokens with a minimum of 0.
Prefill speed (tok/s) – prompt-processing throughput in tokens per second. The default is 4,000 tok/s. The calculator floors a zero or invalid runtime value at 0.0001 tok/s to avoid division by zero.
Decode speed (tok/s) – sequential generation throughput in tokens per second. The default is 60 tok/s and receives the same 0.0001 tok/s floor in the calculation.
Fixed overhead (ms) – delay placed before the first token, measured in milliseconds. The default is 150 ms, the input step is 10 ms, and negative values become 0.
Reading the results
| Printed result | What it means | Practical response |
|---|---|---|
| Total time | Time to first token plus decode time | Compare it with the response-time budget for the whole answer. |
| Time to first token | Fixed overhead plus prompt prefill | Reduce prompt size or measure gateway delay when this is high. |
| Decode time | Output tokens divided by decode speed | Shorten responses or increase measured decode throughput. |
| Prefill time | Prompt tokens divided by prefill speed | Inspect retrieved context and chat-history policy. |
Total time is the end-to-end estimate. It includes the full output even when a streaming interface starts showing text earlier. For an interactive screen, Time to first token is often the earlier signal to monitor because it is the period in which the user sees no response.
The code labels Decode time with the selected output-token count and labels Prefill time with the selected prompt-token count. Those labels help catch a common input error: copying output throughput into the prefill field or entering a character count as tokens.
Warning: the calculator has no queueing, network transfer, moderation, tool execution, cache-hit, or concurrency term.
If prefill time dominates, trim repeated context, reduce retrieved passages, or inspect cache behavior in the actual server. If decode time dominates, response caps have a direct and linear effect. 500 tokens at 60 tok/s take 8.33 seconds.
Fixed overhead is visible inside time to first token. A change from 150 ms to 650 ms adds exactly 0.50 seconds to TTFT and total time, while leaving the two throughput-derived times unchanged.
The formula
The calculator converts overhead to seconds, then uses prefillS = promptTokens / prefillTps, ttft = overheadMs / 1000 + prefillS, and decodeS = outputTokens / decodeTps. Its final result is total = ttft + decodeS. Inputs below zero are clamped to zero; the two rates receive a minimum of 0.0001 tok/s.
| Built-in default | Value used by the code |
|---|---|
| Prompt tokens | 2,000 |
| Output tokens | 500 |
| Prefill speed | 4,000 tok/s |
| Decode speed | 60 tok/s |
| Fixed overhead | 150 ms = 0.15 s |
With the defaults, prefill is 2,000 / 4,000 = 0.50 s. TTFT is 0.15 + 0.50 = 0.65 s. Decode is 500 / 60 = 8.333... seconds, so total time is 0.65 + 8.333... = 8.983... seconds. The display rounds that to 0.50 s prefill, 0.65 s TTFT, 8.33 s decode, and 8.98 s total.
Info: the source calculates an effective token-rate value internally, but the current interface does not print it as a result tile.
This model assumes the selected prefill and decode speeds remain constant through one request. It is a clean way to compare request shapes, although a live service can move those rates when hardware, batching, model routing, or context length changes.
Worked examples
Short support reply. Use 400 prompt tokens, 120 output tokens, 4,000 tok/s prefill, 60 tok/s decode, and 150 ms overhead. Prefill is 0.10 s; TTFT is 0.25 s; decode is 2.00 s; total is 2.25 s. Set a 120-token response limit if that answer form is sufficient.
Retrieved help answer. With 6,000 prompt tokens, 400 output tokens, 3,000 tok/s prefill, 50 tok/s decode, and 200 ms overhead, prefill is 2.00 s, TTFT is 2.20 s, decode is 8.00 s, and total is 10.20 s. Compare that result with a smaller retrieval set before changing hardware.
Detailed analyst response. Enter 3,000 prompt tokens, 1,200 output tokens, 4,000 tok/s prefill, 80 tok/s decode, and 100 ms overhead. The calculation gives 0.75 s prefill, 0.85 s TTFT, 15.00 s decode, and 15.85 s total. The next action is to decide whether 1,200 generated tokens belong in an interactive request.
Long-context edge case. At 50,000 prompt tokens, 500 output tokens, 2,500 tok/s prefill, 50 tok/s decode, and 300 ms overhead, prefill is 20.00 s, TTFT is 20.30 s, decode is 10.00 s, and total is 30.30 s. 50,000 prompt tokens create a 20.30-second first-token wait. Inspect retrieval, history compression, and request routing before presenting this path as chat-like.
Common mistakes
Using words or characters as prompt tokens distorts the estimate. Measure token counts with the tokenizer used by the deployed model, then enter that token figure.
Putting total model throughput in both speed fields assumes prefill and decode behave alike. Record separate measurements because the simulator explicitly models two phases.
Reading Total time as the first visible response produces a pessimistic streaming estimate. Use Time to first token for the initial wait and Decode time for the remaining generation period.
Danger: a rate of zero becomes 0.0001 tok/s in the formula, producing an extremely large time instead of a valid zero-speed result.
Replace a zero rate with a measured nonzero value, or treat the result as a signal that the configuration cannot produce the requested response.
FAQ
Does streaming change the total-time result?
No. The displayed total still includes all generated tokens, even if the client renders them one by one.
Streaming changes perceived progress after TTFT. With 500 output tokens at 60 tok/s, generation lasts 8.33 seconds after the first token appears.
Why does a longer prompt affect first-token time?
The formula processes prompt tokens before the first generated token. It divides prompt size by prefill speed.
At 4,000 tok/s, adding 4,000 prompt tokens adds 1.00 second of prefill time; overhead is unchanged.
Can I enter a custom model speed?
Yes. Both speed fields are numeric inputs, so they accept the measured tok/s figures for a particular deployment.
A value of 0 is not useful for planning because the code substitutes 0.0001 tok/s. Use the actual lower bound you observed instead.
What does fixed overhead include?
It is a manual delay added before the first token. The calculator does not split it into network, application, or provider components.
The default 150 ms becomes 0.15 seconds. Raising it to 1,150 ms adds exactly one second to TTFT and total time.
Why is there no concurrent-user input?
This calculator estimates one request under the supplied throughput. It does not model contention or queues.
Use throughput measurements taken under expected load when planning a service with parallel requests, then compare its TTFT percentile with this single-request baseline.
Disclaimer
This is an educational estimate for planning. Results depend on the model, provider, hardware, configuration, and current pricing, all of which change.
The live calculator on the page is the source of truth for its exact fields. Verify against current prices and a small real test before scaling spend.
