The Tokens per Second Estimator estimates memory-bound language-model decode speed from model size, quantization, memory bandwidth, and efficiency. It also converts that rate into milliseconds per token and time for a requested output length.
It models decoding, where the code assumes every generated token reads the model weights once. Prompt prefill and mixture-of-experts active-parameter effects are outside this estimate.
How to use it
Enter model parameters in billions, then select FP16, INT8, or INT4. The default is an 8B model at INT4.
Select a listed hardware bandwidth or Custom. Listed hardware includes RTX 3060 at 360 GB/s, RTX 4090 at 1,008 GB/s, RTX 5090 at 1,792 GB/s, A100 40GB at 1,555 GB/s, A100 80GB at 2,039 GB/s, H100 SXM at 3,350 GB/s, Apple M3 Max at 400 GB/s, and CPU DDR5 at 80 GB/s.
Check the bandwidth and efficiency against the serving configuration, since the rate is a decode estimate.
When Custom is selected, supply Custom bandwidth in GB/s. Otherwise the interface displays the selected bandwidth.
Set efficiency and output tokens. Defaults are 70% and 500 tokens.
Fields explained
Parameters (billions) – model parameter count, default 8, step 0.1.
Quantization – FP16 uses 16 bits, INT8 uses 8 bits, and INT4 uses 4 bits per parameter.
Hardware (memory bandwidth) – selected device bandwidth, default RTX 4090 at 1,008 GB/s; Custom exposes a 1 GB/s-step input.
Efficiency (%) – the usable share of bandwidth, default 70%, with the code accepting nonnegative values.
Output tokens to generate – decode length, default 500 tokens.
Reading the results
| Result | Meaning | Action |
|---|---|---|
| Speed | Estimated decode tok/s | Compare hardware or quantization choices. |
| Per token | 1,000 divided by tok/s | Use for streaming cadence. |
| Time for output | Output tokens divided by tok/s | Compare with a response cap. |
The final tile also prints model size in GB. It is model parameters multiplied by bits divided by eight.
An 8B INT4 model is 4.0 GB in this calculation.
Warning: a zero model size or zero effective bandwidth yields zero speed and blank dash outputs for time.
Higher bandwidth and efficiency increase the estimate linearly. More parameter bytes decrease it linearly.
The formula
The source computes modelGB = parametersB × (bits / 8), tps = bandwidthGBs × (efficiency / 100) / modelGB, msPerTok = 1000 / tps, and totalS = outputTokens / tps.
| Quantization | Bits |
|---|---|
| FP16 | 16 |
| INT8 | 8 |
| INT4 | 4 |
Default inputs give model size 8 × (4 / 8) = 4 GB. Speed is 1008 × 0.70 / 4 = 176.4 tok/s. One token takes 5.67 ms and 500 tokens take 2.83 seconds.
Info: the code describes typical efficiency as about 60% to 80%, while accepting any nonnegative input.
The result has no prompt processing time, so it should not be read as full request latency.
Worked examples
Small local model. A 3B INT4 model on 400 GB/s hardware at 70% is 1.5 GB. Speed is 186.7 tok/s, 5.4 ms per token, and 500 tokens take 2.7 s. Check memory fit next.
8B INT8 server. At 1,008 GB/s and 70%, model size is 8 GB and speed is 88.2 tok/s. A 500-token answer takes 5.7 s. Compare INT4 quality before changing GPU.
70B INT4 deployment. At 3,350 GB/s and 70%, size is 35 GB and speed is 67.0 tok/s. 1,000 tokens take 14.9 s. Set an output limit for interactive use.
CPU edge case. An 8B FP16 model at 80 GB/s and 70% is 16 GB, so speed is 3.5 tok/s and 500 tokens take 142.9 s. The code estimates 285.7 ms for each token. Use this as a capacity warning.
Common mistakes
Using total VRAM as bandwidth is incorrect. Enter GB/s bandwidth, not GB memory capacity.
Using parameter billions as GB skips the quantization term. Let the chosen bit depth set model bytes.
Adding prefill to this decode-only result inflates the displayed output time. Measure prompt latency separately.
Danger: the model ignores MoE active-parameter effects, so a sparse model can differ from this dense-weight assumption.
Validate with a short generation test on the actual stack.
FAQ
Does the result include prompt time?
No. The source says it ignores prompt or prefill time.
Use a separate prefill measurement for TTFT.
What does Custom do?
It replaces the selected device bandwidth with your number.
The default custom value is 1,000 GB/s.
Why does INT4 increase speed?
It reduces model bytes per parameter in the formula.
INT4 uses 4 bits while FP16 uses 16 bits.
Can efficiency exceed 100%?
The UI accepts nonnegative efficiency values.
Use measured realistic values; the note cites about 60% to 80% typical.
Is this a benchmark?
No. It is a bandwidth-derived estimate.
Kernel choice, batching, and hardware can change measured tok/s.
Disclaimer
This is an educational estimate for planning. Results depend on the model, provider, hardware, configuration, and current pricing, all of which change.
The live calculator on the page is the source of truth for its exact fields. Verify against current prices and a small real test before scaling spend.
