LLM infrastructure
Estimate GPU memory, concurrent requests, tokens per second, and batch runtime for a large language model on your hardware.
1 GPU · 200 generated tokens / request
Capacity is capped by KV memory and the engine sequence limit. Active requests also depend on job size and target concurrency.
Weights use 2, 1.08, or 0.56 bytes per parameter for BF16, FP8, or 4-bit weights respectively. For GQA and MQA, KV bytes per token per GPU are layers × 2 × local KV heads × head dimension × bytes per element. Tensor parallelism shards KV heads when possible and replicates them otherwise. MLA uses a compressed cache on every tensor-parallel GPU. Input and all generated tokens are rounded up to a 16-token cache block for capacity.
Each phase uses the larger of estimated compute time and memory transfer time. Prefill processes input tokens in a wave and produces the first token. Decode reuses the weights across the active batch and reads KV at the average context length. MoE models use the active parameter count for compute and assumes independent, uniform expert routing to estimate weight reads.
Output throughput including prefill divides generated tokens (answer plus thinking) by full-wave duration. Batch time adds consecutive waves, including a smaller final wave when needed. Replicas get balanced workloads.
This is a GPU-resident text generation model for identical GPUs and tensor-parallel replicas. Queueing, CPU offload, expert parallelism, prefix reuse, and continuous batching are outside the model. Engine kernels, quantized checkpoints and interconnects can materially change results. Use the custom GPU option if reported VRAM differs from the nominal preset. Validate with vLLM serving benchmarks on representative prompts. For Qwen YaRN deployments, enter the deployed context limit in a custom model.