This tool predicts end-to-end inference latency for LLM requests across non-streamed and streamed response modes. It evaluates Time to First Token (TTFT), token decoding speed, server queue wait times, network round-trip overhead, and estimated p90 latency variations. Performance engineers use it to design low-latency user interfaces.
Loading calculator...
User-perceived latency in AI applications depends heavily on streaming configuration. While full non-streamed responses force users to wait for the complete text generation cycle, streaming begins rendering text as soon as the first token arrives.
How to use it
Enter your API endpoint’s Time to First Token in milliseconds, representing prompt prefill processing time. Input expected output token count for the response.
Set decode generation speed in tokens per second per stream. Input server queue wait time in milliseconds to account for heavy load conditions.
Measure client-to-server network latency using ping or HTTP ping tests to populate realistic network round-trip metrics.
Set network round-trip time in milliseconds. The dashboard displays full non-streamed response time, perceived first-token streamed latency, raw decode generation time, and estimated p90 latency during busy hours.
Fields explained
Time to first token (ms) – time in milliseconds required for the model to process prompt context and generate token zero. Default value is 400, step size 1.
Output tokens – total count of generated response tokens. Default value is 500, step size 1.
Decode speed (tokens/sec) – single-stream generation throughput speed in tokens per second. Default value is 60, step size 1.
Queue wait (ms) – server-side queueing delay in milliseconds experienced during peak traffic load. Default value is 0, step size 1.
Network round-trip (ms) – client-to-server network latency delay in milliseconds. Default value is 80, step size 1.
Reading the results
| Latency Performance Metric | Time Unit Output | User Experience Impact |
|---|---|---|
| Full response | Total seconds / milliseconds for complete generation in non-streamed mode. | Controls total wait time for batch APIs and non-streaming applications. |
| First token (streamed) | Perceived latency in milliseconds before text starts rendering on screen. | Determines perceived application speed for interactive chat UIs. |
| Decode time | Raw generation time spent executing autoregressive decoding steps. | Identifies how output length impacts total processing duration. |
| Rough p90 | Estimated 90th percentile response time during busy peak traffic windows. | Establish SLA targets for application performance monitoring. |
Autoregressive decoding scales linearly with output token length. Streaming text drops perceived user latency down to the initial first-token arrival time.
Non-streamed API endpoints processing 500 output tokens subject users to multi-second delays before displaying any text.
Enabling response streaming transforms perceived application responsiveness. Enabling streaming reduces perceived user latency from 8.81 seconds down to 480 milliseconds.
The formula
Decode time in milliseconds divides output token count by decode tokens per second, multiplying by 1,000. Base overhead sums queue wait, network round-trip, and TTFT. Full response latency adds base overhead to decode time. Streamed first-token latency equals base overhead. Rough p90 estimates peak latency variance (1.5× base overhead + 1.4× decode time).
The mathematical representation for decode duration and total latency is:
DecodeMs = (OutputTokens / DecodeTokensPerSec) × 1000
BaseOverheadMs = QueueWaitMs + NetworkMs + TtftMs
FullResponseMs = BaseOverheadMs + DecodeMs
The mathematical representation for streamed first-token arrival and p90 estimations is:
StreamFirstTokenMs = BaseOverheadMs
P90EstimateMs = (BaseOverheadMs × 1.5) + (DecodeMs × 1.4)
| Response Mode | Perceived User Latency | Primary Architectural Fit |
|---|---|---|
| Non-Streamed API | Full Generation Time (~8.8s for 500 tokens) | Background agents, webhooks, JSON data extraction pipelines |
| Streamed Server-Sent Events | First Token Time (~480ms) | Interactive AI chat UIs, live coding assistants, co-pilots |
Time to First Token (TTFT) increases with longer input prompt contexts due to the computational complexity of transformer prefill passes.
For a baseline setup with 400ms TTFT, 500 output tokens, 60 tok/s decode speed, 0ms queue, and 80ms network: Base overhead equals 0 + 80 + 400 = 480ms. Decode time equals (500 / 60) × 1,000 = 8,333ms (8.33s). Full response time equals 480 + 8,333 = 8,813ms (8.81s). Streamed first-token latency equals 480ms. P90 estimate equals (480 × 1.5) + (8,333 × 1.4) = 12,386ms (12.39s).
Worked examples
Interactive Consumer Chat UI
A chat product targets low user latency. Parameters: 250ms TTFT, 300 output tokens, 80 tok/s decode speed, 0ms queue, 50ms network. Base overhead: 250 + 0 + 50 = 300ms. Decode time: (300 / 80) × 1,000 = 3,750ms (3.75s). Full response: 4,050ms (4.05s). Streamed first token: 300ms. P90 estimate: 5,700ms (5.70s). Streaming delivers instant feedback within 300 milliseconds.
High-Load Enterprise API Endpoint
An enterprise endpoint experiences server queuing under heavy traffic load. Parameters: 800ms TTFT, 600 output tokens, 40 tok/s decode speed, 1,500ms queue wait, 120ms network. Base overhead: 1,500 + 120 + 800 = 2,420ms (2.42s). Decode time: (600 / 40) × 1,000 = 15,000ms (15.00s). Full response: 17,420ms (17.42s). Server queueing and slow decode speeds drive full response latency to 17.42 seconds (p90 = 24.63s). The team adds GPU capacity to lower queue times.
Fast Local Model Voice Agent
A voice AI agent requires ultra-fast turn-taking. Parameters: 120ms TTFT, 100 output tokens, 120 tok/s decode speed, 0ms queue, 30ms network. Base overhead: 120 + 0 + 30 = 150ms. Decode time: (100 / 120) × 1,000 = 833ms (0.83s). Full response: 983ms (0.98s). Streamed first token: 150ms. P90 estimate: 1,391ms (1.39s). The system achieves sub-second full responses for voice conversations.
Background Document Summarization Worker
A background worker summarizes dense PDFs. Parameters: 1,200ms TTFT, 1,000 output tokens, 50 tok/s decode speed, 500ms queue, 100ms network. Base overhead: 500 + 100 + 1,200 = 1,800ms (1.80s). Decode time: (1,000 / 50) × 1,000 = 20,000ms (20.00s). Full response: 21,800ms (21.80s). Because the worker runs asynchronously in the background, the 21.8-second wait presents zero user interface issues.
Common mistakes
Evaluating UI responsiveness based on full response generation time instead of first-token arrival leads to unnecessary optimization work. Enabling Server-Sent Events (SSE) streaming improves perceived user speed instantly without altering backend hardware.
Failing to account for server queueing delays under load distorts SLA benchmarks. Latency benchmarks measured during quiet off-peak hours degrade significantly when incoming request volume creates server queues.
Ignoring prompt length impact on TTFT creates performance surprises. Large prompt payloads (such as 50,000 RAG tokens) increase prefill compute time, elevating TTFT from 200ms to several seconds before text generation begins.
Deploying interactive chat interfaces without response streaming forces users to stare at blank screens for several seconds during generation.
Implement response streaming on all interactive UI endpoints to minimize perceived user latency.
FAQ
What is Time to First Token (TTFT) and why does it matter?
TTFT measures the time elapsed between sending an API request and receiving the very first generated text token. It encompasses network transit, server queueing, and prompt prefill processing.
TTFT determines how fast an application feels to an end user when streaming is enabled.
How does response streaming improve user experience?
Response streaming uses HTTP Server-Sent Events (SSE) or WebSockets to transmit generated tokens to the client in real time as they decode. Instead of waiting for full completion, users read text immediately.
Streaming reduces perceived latency from full generation time down to TTFT.
Why does decode time scale linearly with output token length?
Autoregressive language models generate text one token at a time, feeding each generated token back into the model to predict the next token. Generating 500 tokens requires 500 sequential forward passes.
Doubling output token length doubles raw generation decode time.
How does prompt context size affect prefill latency (TTFT)?
Before generating token zero, the model must compute attention key-value states for all input prompt tokens simultaneously (the prefill phase). Larger input prompts require more prefill compute.
Ingesting massive 100,000-token prompts increases TTFT significantly compared to short 500-token prompts.
What is p90 latency and why is it useful for SLA monitoring?
P90 latency represents the response time threshold below which 90 percent of all requests complete. It captures tail latency spikes caused by server queueing, large outputs, or network jitter.
P90 latency provides a realistic performance benchmark for busy operational periods.
Disclaimer
This tool predicts inference latency and p90 variations based on simplified mathematical models and user-entered performance parameters. Actual response times depend on cloud provider server load, dynamic GPU batching, network routing stability, model parameter precision, and client rendering performance.
The interactive calculator on this page serves as the primary resource for testing latency scenarios and UI design planning. Performance engineering teams should conduct empirical latency telemetry logging across production API endpoints to measure true TTFT and decode distribution statistics.
Been using this calculator to model latency for a distributed inference setup I’m building for my thesis. The TTFT vs decode speed breakdown is genuinely helpful because I kept conflating them in my head. Real question though: the p90 estimate formula (1.5x base overhead + 1.4x decode time) seems arbitrary. What’s the justification for those multipliers? I tested it against actual production logs from an API endpoint and got wildly different variance patterns depending on model size. For a 7B parameter model, p90 was closer to 1.2x the formula’s estimate, but with a 70B model it overshot by 30%. Are those multipliers based on specific hardware configurations or load profiles? Would help a lot if there was a way to adjust them based on your actual infrastructure instead of using one-size-fits-all constants.
Great question on the p90 multipliers. You’re right to be skeptical about those constants. The 1.5x and 1.4x values are empirical approximations derived from typical datacenter load patterns during peak hours, but they’re absolutely not universal. The variance you’re seeing makes sense: smaller models like 7B tend to have more predictable latency curves because they fit in fewer GPUs and experience less queueing variance, while 70B models spread across multiple devices accumulate more unpredictable network and synchronization overhead. For your thesis work, I’d recommend logging actual p50, p95, and p99 latencies from your inference runs and deriving local multipliers specific to your hardware. If you’re using vLLM or TensorRT, you can extract these metrics from the logs pretty easily. The calculator works best as a quick sanity check rather than a production SLA tool without calibration to your actual infrastructure.
Thanks for the practical advice. I actually do have vLLM logs from my test runs, so I can extract those actual percentiles and recalibrate. That makes a lot of sense. The batching point is really useful too because I was assuming fixed queueing delays but dynamic batching is going to throw that off completely.
Perfect, vLLM logs are ideal for this. When you pull the percentile data, pay particular attention to how queue wait scales with batch size. You’ll likely see non-linear growth, especially as you approach GPU memory limits. Dynamic batching introduces latency spikes at certain utilization thresholds that the linear p90 formula can’t capture. If you find that pattern in your logs, it’s worth feeding those observed percentiles back into the calculator as a reality check. Also worth noting: if you’re testing different quantization schemes (like GPTQ or AWQ for that 70B model), TTFT can shift noticeably even though decode speed stays relatively consistent. Good luck with the thesis.
Got early access to this last month and it’s genuinely miles ahead of the latency estimation tools floating around on GitHub. The streaming vs non-streamed comparison finally makes visual sense. I’ve been comparing this to the latency calculator from Together AI’s docs and this one handles queue wait modeling way better. The network round-trip field is clutch because most tools ignore that entirely. Only thing: I wish it showed per-layer breakdown for different model architectures since TTFT varies so much between a Llama 3 8B and a Mixtral. Still, for quick ballpark estimates before spinning up infrastructure, this saves hours.
Thanks for the comparison to Together AI’s approach. You’re spot on about queue modeling being the differentiator. Most calculators treat queueing as a fixed offset, but real-world behavior depends heavily on batching strategy and whether you’re using dynamic batching or continuous batching like vLLM. Regarding per-layer breakdown: that’s a known limitation right now. The tool currently uses aggregate TTFT, which works fine for rough estimates but loses granularity. If you need architecture-specific predictions, the best workaround is benchmarking TTFT directly on your target hardware. For Llama 3 8B, TTFT typically lands around 200-300ms on a single H100, but Mixtral’s sparse MoE routing adds unpredictable overhead. That said, if you run your actual models through this calculator with measured TTFT values from your specific setup, the downstream estimates become quite accurate.