Training time estimator for GPU budget planning

The Training Time Estimator turns a model size, training-token target, GPU count, throughput, utilization, and hourly rate into a compute-only training schedule and rental estimate. It uses the dense-transformer 6ND compute rule, which makes the assumptions visible before a training run is booked.

The calculator reports wall-clock days and hours, total ZFLOPs, GPU-hours, and estimated GPU rental cost. It is useful for comparing an eight-GPU run with a larger cluster, or for checking whether a planned dataset and model size fit a deadline.

How to use it

Set Model parameters in billions and Training tokens in billions. The defaults are 7B parameters and 300B training tokens. These values are multiplied by one billion inside the formula, so do not enter the full raw count.

Enter the Number of GPUs and the peak TFLOP/s per GPU. The default configuration is eight GPUs at 312 TFLOP/s each. The hint names A100 at about 312 BF16 TFLOP/s and H100 at about 990.

Check that peak throughput uses the same precision and hardware mode planned for training.

Set Model FLOPs utilization, abbreviated MFU. The interface starts at 40%, and its own note places real training around 30% to 50% of peak. Then enter GPU cost per hour, which defaults to $2.50.

Read Training time as wall-clock time, then use GPU-hours and cost to compare reservations. The code lets GPU count fall no lower than one during calculation, even if zero is entered.

Fields explained

Model parameters (billions) – dense model size in billions of parameters. Default: 7; input step: 0.1B.

Training tokens (billions) – tokens presented during training, expressed in billions. Default: 300B; input step: 1B.

Number of GPUs – accelerators used in the aggregate calculation. Default: 8; input step: 1; runtime minimum: 1.

Peak TFLOP/s per GPU – theoretical throughput at the selected precision. Default: 312 TFLOP/s; input step: 1.

Model FLOPs utilization (%) – the fraction of peak throughput achieved by the training job. Default: 40%; the calculation clamps it between 1% and 100%.

GPU cost / hour – one GPU-hour price in dollars. Default: $2.50; input step: $0.01.

Reading the results

Adding GPUs lowers the displayed wall-clock time because aggregate throughput rises. GPU-hours remain approximately unchanged for the same MFU, since fewer hours are multiplied by more GPUs. Real distributed efficiency can alter that relationship.

Compute is shown in ZFLOPs, where one ZFLOP is 10 to the 21st power floating-point operations. The default 7B by 300B run is 12.60 ZFLOPs.

Warning: the cost tile is GPU rental only. It does not add storage, data preparation, engineer time, evaluations, or failed runs.

MFU has a direct effect. Raising it from 40% to 50% makes effective per-GPU throughput 25% higher and cuts the same compute estimate by 20%.

The formula

The source uses flops = 6 × N × D, where N is parameters and D is training tokens, both converted from billions. Effective per-GPU throughput is peakTFLOPs × 10^12 × MFU/100. Aggregate throughput multiplies that value by GPU count. Seconds are flops / aggregate; hours are seconds divided by 3,600; GPU-hours are hours multiplied by GPU count; cost is GPU-hours multiplied by hourly price.

For the defaults, compute is 6 × 7e9 × 300e9 = 1.26e22 FLOPs. Effective aggregate throughput is 312e12 × 0.40 × 8 = 9.984e14 FLOP/s. That produces 12,620,192 seconds, or 3,506 hours and 146.1 days. GPU-hours are 28,045; cost is $70,112.

Info: the source calls 6ND a first-order estimate for dense transformers and warns that MoE, long-context attention, and data stalls change it.

The displayed days round to one decimal, while hours, GPU-hours, and dollars round to whole values. Preserve more precision in a procurement sheet if two options are close.

Worked examples

Small experiment. Use 1B parameters, 10B tokens, two 312-TFLOP/s GPUs, 40% MFU, and $2.50/hour. Compute is 0.06 ZFLOPs, training takes 66.8 hours or 2.8 days, uses 134 GPU-hours, and costs $334. Schedule the run with room for evaluation.

Default-sized run. At the six defaults, the calculator shows 146.1 days, 12.60 ZFLOPs, 28,045 GPU-hours, and $70,112. Compare that duration with a larger cluster only after checking communication efficiency.

H100 cluster. Enter 7B parameters, 300B tokens, 32 GPUs, 990 TFLOP/s, 45% MFU, and $5/hour. The result is 10.2 days, 12.60 ZFLOPs, 7,856 GPU-hours, and $39,282. The next step is to test whether 45% MFU is observed at 32 GPUs.

Scale case. For 70B parameters, 1,000B tokens, 128 GPUs, 990 TFLOP/s, 40% MFU, and $5/hour, compute is 420 ZFLOPs, time is 95.9 days, GPU-hours are 294,613, and cost is $1,473,064. The compute-only rental estimate is about $1.47 million. Add the source note’s 20% to 40% schedule allowance before setting a date.

Common mistakes

Entering raw token counts instead of billions makes the estimate one billion times too large. Enter 300 for 300B training tokens.

Using peak TFLOP/s as achieved throughput omits MFU. Put the actual utilization measurement or a conservative 30% to 50% planning value in the MFU field.

Reading GPU-hour cost as the full project budget excludes work outside accelerator rental. Maintain a separate line for data, storage, evaluation, and personnel.

Danger: the calculator clamps MFU below 1% to 1%, so a zero entry produces a very long estimate rather than zero compute.

Treat that outcome as invalid input and replace it with a measured or planned utilization rate.

FAQ

Does adding GPUs reduce GPU-hours?

In the formula, more GPUs reduce wall-clock hours while multiplying GPU count by the same amount.

At equal MFU, GPU-hours stay near the same value; real scaling losses can increase them.

What is MFU?

MFU is Model FLOPs utilization, the achieved fraction of peak throughput.

The calculator starts at 40% and notes a real-training range around 30% to 50%.

Does the estimate work for MoE models?

It is presented as a dense-transformer first-order estimate.

The source specifically says MoE can change the figure.

Why is GPU cost not enough for a total budget?

The displayed cost multiplies GPU-hours by the entered rental price only.

Restarts, evaluations, storage, and staff are not in that multiplication.

How much schedule contingency should I add?

The calculator’s note recommends 20% to 40% beyond the compute-only figure.

For a 10-day output, that note implies planning roughly 12 to 14 days.

Disclaimer

This is an educational estimate for planning. Results depend on the model, provider, hardware, configuration, and current pricing, all of which change.

The live calculator on the page is the source of truth for its exact fields. Verify against current prices and a small real test before scaling spend.