For a first large language model deployment, aim for the smallest tested configuration that meets the agreed service targets, with a clear route to expansion. Buying more GPUs can create headroom, but it will not tell you whether the application needs that capacity or where a bottleneck lies.
Define the service before selecting hardware
Start with the application owner. Establish the model revision, weight precision, typical and maximum input/output lengths, expected request arrivals and likely bursts. An internal assistant answering short questions and an overnight document-processing queue need different performance targets.| Measure | What the customer should learn |
| Time to first token | How long users wait before a response begins. |
| Generation rate | How quickly each response continues after it starts. |
| End-to-end latency | How long a complete request takes, including queueing and relevant application steps. |
Measure the user experience as well as the model server. A retrieval step can make the application slow even while GPU utilisation is low. Where users connect from several locations, test those access paths too. Keep estimates labelled so the pilot can replace them with observations.
Budget memory beyond the weights
For dense models, theoretical weight storage in bytes is parameter count multiplied by bits per parameter, divided by eight. An 8B model therefore has approximately 16 GB of BF16 weights, or 4 GB of theoretical four-bit weights, using decimal GB. Quantisation metadata and implementation overhead add to those figures. Check output quality on the intended task before choosing lower precision.Inference also needs memory for the key-value cache, activations and runtime buffers. The cache retains attention information used during generation. Its size depends on the model architecture, cached tokens, active requests and cache precision.
For a conventional transformer, a simplified cache estimate is: 2 × layers × KV heads × head dimension × cached tokens × bytes per element. As a separate arithmetic example, 32 layers, eight KV heads, a head dimension of 128 and a BF16 cache require about 1 GiB for one request retaining 8,192 tokens. Four such requests require about 4 GiB, before other allocations and without shared-prefix savings.
That example does not describe every 8B model. Check the selected model configuration and profile the actual runtime. For AI Infrastructure sizing, the useful figure is the complete memory budget per GPU under the intended workload.
Give the pilot a specific brief
| Requirement | Illustrative assumption |
| Application and model | Internal document assistant; one dense 8B model, exact revision to be selected. |
| Precision | BF16 weights; cache precision recorded separately. |
| Requests | Typical input 2,000 tokens; output 500 tokens; maximum combined sequence 8,192 tokens. |
| Demand | Four simultaneous requests as an initial requirement, not demonstrated capacity. |
| Availability | Single-instance pilot; production resilience assessed separately. |
A 48 GB GPU could be evaluated for this brief. CPU, host RAM, storage and networking still need a sizing basis. Include document extraction, retrieval and any other services sharing the host. For a document assistant, count retrieved text in the prompt length. Otherwise, the sizing test may be much easier than the application workload.
Find the load limit before expanding
Establish a single-request baseline, then increase load using realistic arrival patterns, including bursts. Record the model, runtime version, precision and request mix. Observe first-token and complete-request latency, per-request generation rate, throughput, queue growth and errors. Use a defined percentile such as p95 where appropriate, and include slow and failed requests in the review.The practical capacity of your AI Infrastructure is the load sustained while meeting the service targets. Higher aggregate throughput can come with longer waits for individual users. A faster-looking throughput result may therefore be the wrong setting for an interactive service.
If one GPU holds the model comfortably but cannot serve enough requests, additional replicas may help. If the model does not fit, splitting it across devices requires runtime support and suitable interconnects. A slow retrieval service or CPU queue needs a different remedy. Availability is another design decision: unused GPU memory does not provide failover.
Turn the pilot into an operating plan
For a planning review with advange, bring the workload brief, sizing assumptions and complete pilot results. Agree the relevant deployment, acceptance and operational work against the existing environment and supplier responsibilities.The handover should identify tested capacity, software versions, monitoring thresholds and the conditions that trigger another sizing review. A first deployment cannot predict every future requirement. It can give the business a measured starting point for its AI Infrastructure and a clear reason for the next investment.
