Quick Take
Quick Take: An 8x H200 server holds 8 x 141 GB = 1,128 GB, calculated from NVIDIA published specs. One trillion parameters need about 1,000 GB at FP8, 500 GB at 4 bits and 2,000 GB at BF16, so only 4-bit leaves room. BENQUAN GLOBAL LIMITED supplies AI servers and sources NVIDIA through authorized channels.
You want to serve a 1-trillion-parameter model on a single 8x H200 node (1,128 GB HBM3e) instead of buying a multi-node InfiniBand fabric. Weight quantization makes it fit on paper, but KV-cache and activation overhead at production concurrency can push you straight into OOM.
3-Line Fast RFQ Template (Ref: BQ-H200-1T) — Copy and email to info@benquan.hk or open pre-tagged RFQ form →
Ref: BQ-H200-1T (article-trillion-parameter-model-8x-h200-server-memory) 1) Model Parameter Size, Quantization (FP8 / INT4) & Context Length: 2) Server Platform & Quantity (8x H200 SXM 141GB vs 2-node cluster): 3) Target Delivery Window & Destination:
How Much GPU Memory Does an 8x H200 Server Have?
Quick Answer: NVIDIA lists 141 GB of HBM3e at 4.8 TB/s per H200, and HGX H200 systems come with 4 or 8 GPUs. Eight GPUs give 8 x 141 = 1,128 GB of GPU memory in one NVLink domain, calculated from NVIDIA published specs.
The figures come from the NVIDIA H200 page, checked 2026-10-16: GPU memory 141 GB, memory bandwidth 4.8 TB/s, NVLink 900 GB/s per GPU on the SXM form factor, and HGX H200 server options with 4 or 8 GPUs. NVIDIA marks these as preliminary specifications that may change. System RAM is separate and does not count toward the 1,128 GB.
For the 8-GPU HGX configuration, see the NVIDIA H200 GPU server page and the AI compute overview.
How Much Memory Do 1 Trillion Parameters Need at FP8, 4-Bit and BF16?
Quick Answer: Weights need parameters multiplied by bytes per parameter: about 2,000 GB at BF16, 1,000 GB at FP8 and 500 GB at 4 bits for one trillion parameters. Only the 4-bit case leaves meaningful room in 1,128 GB. These are calculated from NVIDIA published specs plus the standard bytes-per-parameter arithmetic, not measured results.
| Precision | Bytes per parameter | Weights for 1T parameters | Versus 1,128 GB |
|---|---|---|---|
| BF16 | 2 | 1,000,000,000,000 x 2 = 2,000 GB (about 2 TB) | 872 GB short; needs multi-node |
| FP8 | 1 | 1,000,000,000,000 x 1 = 1,000 GB (about 1 TB) | 128 GB left (11.3 percent) for KV cache and buffers |
| 4-bit weights | 0.5 | 1,000,000,000,000 x 0.5 = 500 GB (about 0.5 TB) | 628 GB left |
Method: 1 trillion parameters is 1,000,000,000,000. One byte per parameter is 10^12 bytes, which is 1,000 GB in decimal units. The table counts weights only. The KV cache, activations, framework buffers and any quantization scale tables come on top, so real headroom is smaller than shown. NVIDIA lists FP8 and INT8 Tensor Core rates for the H200 but no FP4 rate, so a 4-bit layout is a software weight-only choice to validate for accuracy and kernel support.
This page does not state any specific model's parameter count; use the model publisher's own documentation for that.
Does a Mixture-of-Experts Model Change the Memory Math?
Quick Answer: It changes compute, not weight memory. A mixture-of-experts model routes each token to a subset of experts, so fewer parameters do work per token, but the full set of expert weights still has to sit in GPU memory or be offloaded at a speed cost.
Size memory on total parameters and compute on active parameters. A trillion-parameter mixture-of-experts model therefore needs the same weight footprint as a dense one of equal total size, while its per-token compute is lower. Expert routing also adds communication, which is one reason NVIDIA positions rack-scale NVLink systems for such models.
When Do I Need Multi-Node or NVL72 for a Trillion-Parameter Model?
Quick Answer: Move beyond one 8x H200 node when weights plus KV cache exceed 1,128 GB. At BF16 the weights alone are about 2,000 GB, so at least two 8-GPU nodes are needed: 2,000 / 141 = 14.2, so 15 GPUs minimum for weights alone, which in practice means two 8-GPU nodes (16 GPUs).
Two 8-GPU nodes give 2 x 1,128 = 2,256 GB, which covers 2,000 GB of BF16 weights with 256 GB left. Multi-node adds network traffic between nodes, which is slower than the 900 GB/s NVLink inside one HGX board, so design the interconnect before the purchase order.
NVIDIA describes the GB200 NVL72 as 72 Blackwell GPUs in one NVLink domain with 13.4 TB of HBM3E. That is a different platform generation and a rack-scale, liquid-cooled design; evaluate it when you need one NVLink domain larger than eight GPUs, not as a drop-in replacement for an H200 server.
Is Inference the Same as Training for Memory?
Quick Answer: No. The weights-only figures above describe inference at rest. Training also stores gradients and optimizer states, which multiply memory use well beyond the weight footprint, so a trillion-parameter training run is a multi-node cluster problem rather than a single 8-GPU server problem.
Use one 8x H200 server for inference or fine-tuning of smaller or heavily compressed models, and size training clusters from the framework's memory model and your target throughput. Ask for the exact HGX configuration, interconnect and support terms on the RFQ; the HGX H200 8-GPU server RFQ in 12 lines shows the line items to specify.
RFQ Checklist
Quick Answer: State the model's total parameters and precision, context length and workload type, so GPU count, memory and interconnect are sized on one brief. Quotations written on identical terms can then be compared line by line, avoiding mismatched memory assumptions between suppliers.
What to put on the RFQ
- Target model total parameters and precision (BF16, FP8 or 4-bit) taken from the publisher's documentation
- Context length and concurrent users, so KV cache can be sized
- Inference, fine-tuning or training
- HGX H200 8-GPU configuration, CPU, system RAM and NVMe
- Interconnect for multi-node (InfiniBand or Ethernet) and node count
- Delivery country, support years and quote validity window
See the NVIDIA H200 GPU server page, then submit the brief on rfq.html.
Sources
Quick Answer: Every NVIDIA figure on this page comes from NVIDIA's own H200 and GB200 NVL72 pages, checked 2026-10-16. NVIDIA marks H200 specifications as preliminary and subject to change, so confirm them for your exact configuration. All sizing arithmetic is calculated from NVIDIA published specs, not measured.
Frequently Asked Questions
Quick Answer: An 8x H200 server has 1,128 GB of GPU memory, calculated from NVIDIA published specs. A trillion-parameter model fits at 4-bit, barely at FP8 and not at BF16 on a single node. Confirm the model's published size, precision and context length before sizing.
Only with compression. Calculated from NVIDIA published specs, 8 GPUs at 141 GB give 1,128 GB. One trillion parameters need about 1,000 GB at FP8 (1 byte each), about 500 GB at 4 bits and about 2,000 GB at BF16. So 4-bit fits comfortably, FP8 fits with almost no room for KV cache, and BF16 needs more than one node.
NVIDIA lists 141 GB of HBM3e per H200 GPU. Eight GPUs on an HGX board therefore total 8 x 141 = 1,128 GB, calculated from NVIDIA published specs.
Very little. 1,128 GB minus 1,000 GB of FP8 weights leaves 128 GB, about 11 percent, before runtime buffers, activations and the KV cache are counted. Long contexts or many concurrent users are likely to exceed that.
NVIDIA's H200 specification table lists FP8 and INT8 Tensor Core rates and no FP4 rate. A 4-bit weight format is therefore a software and quantization question; confirm the framework, kernels and accuracy impact before sizing around it.
No. A mixture-of-experts model activates only some experts per token, which reduces compute, but all expert weights normally have to be resident or offloaded. Memory is sized on total parameters, not active parameters.
When weights plus KV cache exceed the 1,128 GB of one node, or when you need to train. A BF16 trillion-parameter model needs about 2,000 GB of weights, so at least two 8-GPU nodes. NVIDIA describes GB200 NVL72 as a 72-GPU NVLink domain with 13.4 TB of HBM3E.
Key Takeaway
Key Takeaway: Calculated from NVIDIA published specs, 1,128 GB fits a trillion parameters at 4 bits, barely at FP8 and not at BF16. Mixture-of-experts does not shrink weight memory, and training needs far more. BENQUAN GLOBAL LIMITED supplies AI servers and sources NVIDIA through authorized channels.