AI / Hardware / GPU Procurement / Inference & Training

GPU selection 2026: does H100 at 80 GB and 3.35 TB/s fit, or do you need H200 at 141 GB?

Procurement guide covering global full-spec GPUs, export-compliant and domestic options, RTX PRO Blackwell workstation cards, and workload-matched selection with NVIDIA-sourced specs.

Quick Take

Quick Take: GPU selection in 2026 adds compliance as the sixth dimension. NVIDIA lists 80 GB at 3.35 TB/s for H100 SXM and 141 GB at 4.8 TB/s for H200 SXM, so memory and bandwidth, not headline TFLOPS, usually decide inference fit. RTX PRO 6000 Blackwell carries 96 GB of GDDR7 ECC on one card. Confirm export-controlled SKU availability in writing first.

Buying training GPUs for an inference workload wastes half your CapEx; buying air-cooled cards for a 70B+ long-context model chokes on memory bandwidth. Match your parameter count, batch concurrency, and facility power cap to the exact GPU tier before freezing your BOM.

Key extractable numbers

Memory-first shortlist from NVIDIA-listed SXM class figures on this page: H100 SXM at 80 GB and 3.35 TB/s, H200 SXM at 141 GB and 4.8 TB/s, and RTX PRO 6000 Blackwell at 96 GB GDDR7 ECC on one card. Treat the selection framework as six dimensions (compute, memory, interconnect, power, price, compliance)—a card that cannot be procured legally scores zero regardless of TFLOPS. This guide quotes no street prices; confirm export-controlled SKU availability in writing before freezing the BOM.

3-Line Fast RFQ Template (Ref: BQ-GPU-SEL) — Copy and email to info@benquan.hk or open pre-tagged RFQ form →

Ref: BQ-GPU-SEL (article-gpu-selection-guide-2026)
1) Workload Type (Training vs Inference), Model Size & Concurrency: 
2) Candidate GPUs & Node Count (H200 / B300 / L40S / RTX PRO 6000): 
3) Power/Cooling Constraint & Target Delivery Month: 

Selection Framework: 6 Dimensions

Quick Answer: Evaluate compute, memory capacity and bandwidth, interconnect (NVLink or PCIe), power, price and, above all, compliance and availability. A card that cannot be procured legally has zero value regardless of specs. Confirm each dimension in a written supplier datasheet or quotation, and treat any figure without a named source as unverified.

This guide quotes no prices. GPU prices move with export policy, supply and channel availability; confirm current pricing (industry estimate, to be confirmed) and compliance status with your supplier before procurement.

6
Selection Dimensions
80GB
H100 SXM Memory (NVIDIA)
141GB
H200 SXM Memory (NVIDIA)
96GB
RTX PRO 6000 Single-Card

For the full platform, fabric, power, and TCO framework, see the GPU Server Buying Guide 2026.

The intuition that drives most decisions below: training is compute-intensive; inference decode is memory-bandwidth-intensive. During decode, each generated token requires reading the model weights from GPU memory, so memory capacity and bandwidth matter as much as TFLOPS.

Global Full-Spec Datacenter GPUs

Quick Answer: NVIDIA lists H100 SXM at 80 GB HBM3 and 3.35 TB/s, and H200 SXM at 141 GB HBM3e and 4.8 TB/s, both with 900 GB/s NVLink and up to 700 W (configurable). B200 and B300 memory is listed in the verified table further below. All of these are subject to export controls.

These cards define the performance baseline. Even if you cannot buy them directly, you need these numbers to calibrate alternative BOMs.

ModelMemoryBandwidthFP16 Tensor (with sparsity)InterconnectMax TDP
H100 SXM80 GB3.35 TB/s1,979 TFLOPSNVLink 900 GB/s, PCIe Gen5 128 GB/sUp to 700 W (configurable)
H200 SXM141 GB HBM3e4.8 TB/s1,979 TFLOPSNVLink 900 GB/s, PCIe Gen5 128 GB/sUp to 700 W (configurable)

Sources: NVIDIA H100 GPU page and NVIDIA H200 GPU page. NVIDIA marks the Tensor Core figures as with sparsity; the H200 figures are labelled preliminary and may change. Checked 2026-10-15.

Compliance notice: Full-spec Hopper and Blackwell data center GPUs are subject to US export controls for many destinations. Grey-market or re-export units carry elevated compliance and warranty risk; confirm status before you commit.

RTX PRO Blackwell Workstation Cards

Quick Answer: NVIDIA lists RTX PRO 6000 Blackwell with 96 GB of GDDR7 ECC memory on a single card. That capacity lets a small team prototype quantized large models without an HGX server; check memory sizing below for your model. Confirm the exact variant, power and cooling needs in the NVIDIA datasheet before ordering.

For small and mid-sized teams, a single high-memory card can replace a multi-GPU server for development and proof-of-concept work. It is a PCIe-class card, so multi-card scaling does not use NVLink the way HGX systems do; confirm the exact variant (Server, Workstation or Max-Q) against the NVIDIA line card and the RTX PRO 6000 Blackwell Edition RFQ Map.

China-Compliant Cards: H20, L20, RTX 6000D

Quick Answer: NVIDIA has built reduced-specification data center and workstation cards for export-controlled markets, but availability follows export policy rather than demand, and these SKUs do not have full public datasheets on NVIDIA data center pages. This guide therefore gives no performance percentages for them.

Export rules for advanced computing chips changed several times between 2022 and 2025, and product lines were created, restricted or withdrawn in response. Read the current rules from the US Department of Commerce (BIS) and the supplier statement for your exact part number rather than from a summary table.

How to evaluate a reduced-spec card

Ask the supplier for the written datasheet of the exact SKU and compare memory capacity, memory bandwidth, interconnect and power against the verified NVIDIA tables on this page. Do not assume a reduced-spec card matches a full-spec card in throughput; request workload-specific test results for your model.

Procurement warning: Before purchasing any export-controlled SKU, verify current license status, channel availability, and whether your organization is subject to domestic procurement guidance.

Domestic GPU Alternatives

Quick Answer: Domestic accelerators such as Huawei Ascend, Hygon DCU and Cambricon are options for compliance-first deployments. Their software stacks differ from CUDA, so budget adaptation effort, and obtain current specifications and benchmarks from the vendor for your model. Request written datasheets and run a pilot first.

Ascend uses the CANN software stack, which is not CUDA-compatible, so operator coverage, debugging tools and framework integrations should be tested on your own models before you commit. This guide quotes no throughput percentages or prices for these parts because it has no vendor-published figures to cite. See the KunLun G8680 V3 Ascend 910 BOM RFQ for chassis-level BOM gaps.

Use-Case Recommendations

Quick Answer: Match the card to the bottleneck: compute-bound training favours full-spec HGX-class GPUs with NVLink; decode-bound inference favours memory capacity and bandwidth; small proof-of-concept work favours a single high-memory card. Always validate against your own model, context length and batch size before freezing the BOM; vendor tables are only a starting point.

ScenarioDirectionRationale
Large-model trainingHGX-class H100/H200/B200 nodesCompute-bound, needs NVLink scale-up
Adapter fine-tuning, small models1-2 high-memory cardsSingle card can suffice with quantization
Online inference, full-specH200 or Blackwell nodesMemory capacity and bandwidth per GPU
Small team proof of conceptRTX PRO 6000 (96 GB)Single card, no HGX server needed
Compliance-first deploymentsDomestic acceleratorsDomestic supply chain; test your software stack first

Memory Sizing Formula

Quick Answer: Inference memory equals parameters times bytes per parameter, plus key-value cache, plus framework overhead. FP16 uses 2 bytes, INT8 one byte, INT4 about half a byte. A 70B model needs about 140 GB for weights at FP16 and about 35 GB at INT4, before cache and overhead.

  • Inference: memory = parameters x bytes-per-value + KV cache + framework overhead. KV cache grows with context length and batch size, and models with grouped-query attention need less.
  • Training (full parameter): weights, gradients and optimizer states multiply the footprint several times over the weights alone; size from your optimizer and precision settings.
  • Training (LoRA/QLoRA): freeze the base model and train low-rank adapters, which cuts memory sharply compared with full training.

8 Common Pitfalls

Quick Answer: 1) Using a reduced-spec card for training without a test. 2) Assuming reduced-spec equals full-spec. 3) Forcing tensor parallelism on cards without NVLink. 4) Buying grey-market full-spec cards without compliance review. 5) Underestimating domestic GPU software adaptation. 6) Single-sourcing export-controlled SKUs. 7) Ignoring power and cooling. 8) Over-procuring before a known architecture transition.

Pitfalls 1-2: Reduced-spec cards

Get the written datasheet and a workload test before you size a cluster on a reduced-spec card. Do not infer training time from another card's results.

Pitfall 3: Multi-card tensor parallelism without NVLink

PCIe-only cards communicate more slowly than NVLink-connected GPUs (900 GB/s per NVIDIA for H100 and H200 SXM versus 128 GB/s PCIe Gen5). Test scaling before you buy multiple cards.

Pitfalls 4 and 6: Compliance and single sourcing

Cards from unofficial channels risk missing warranty and compliance audit findings. Policy can change quickly, so keep a backup plan for production inference.

Pitfall 5: Domestic GPU software adaptation

Operator and framework adaptation takes engineering time. Budget it in the project plan.

Pitfalls 7-8: Power, cooling and timing

NVIDIA lists up to 700 W per H100 or H200 SXM GPU, so an 8-GPU node draws 5.6 kW from GPUs alone (8 x 700 W) before CPUs, fans and networking. Next-generation parts can reduce the value of current inventory; avoid over-procuring just before a known transition.

Verified Memory and Bandwidth Specs from NVIDIA

Quick Answer: NVIDIA documentation lists 141 GB HBM3e at 4.8 TB/s for H200, 180 GB for B200 and 288 GB for B300, both B200 and B300 at up to 8 TB/s per GPU. NVIDIA lists RTX PRO 6000 Blackwell with 96 GB of GDDR7 ECC memory.

ItemH200 SXMB200 SXMB300 SXMRTX PRO 6000 Blackwell
Memory per GPU141 GB HBM3e180 GB HBM3e288 GB HBM3e96 GB GDDR7 ECC
Memory per 8-GPU node1.1 TB1.44 TB2.30 TBNot applicable
Memory bandwidth4.8 TB/sUp to 8 TB/sUp to 8 TB/sSee NVIDIA line card

Sources: NVIDIA HGX AI Factory reference architecture, NVIDIA H200 page and the NVIDIA professional graphics line card. Memory sizing: a 70B-parameter model at 16-bit precision needs about 140 GB for weights alone (70 billion x 2 bytes), before cache and activations, so it does not fit one 96 GB card without quantization. Checked 2026-10-15.

How to Shortlist a GPU in Five Steps

  1. Step 1: Write the model size and context length, then compute the memory you need per GPU.
  2. Step 2: Compare per-GPU memory and bandwidth in the NVIDIA table above, because inference is often bandwidth bound.
  3. Step 3: Check interconnect needs: NVLink for multi-GPU training, PCIe for single-card inference.
  4. Step 4: Check power and cooling at rack level, including direct liquid cooling for the newest systems.
  5. Step 5: Confirm availability and compliance status in writing before you raise the purchase order.

Frequently Asked Questions

Quick Answer: Memory capacity, memory bandwidth, interconnect and compliance decide most GPU choices; NVIDIA lists 141 GB and 4.8 TB/s for H200 SXM and 96 GB for RTX PRO 6000 Blackwell. Do not rely on unsourced throughput percentages or price ratios; verify everything against written vendor datasheets.

NVIDIA lists 141 GB HBM3e for H200, 180 GB for B200 and 288 GB for B300 per GPU, and 96 GB GDDR7 ECC for RTX PRO 6000 Blackwell. NVIDIA lists 80 GB for H100 SXM. Checked 2026-10-15.

It depends on model size and availability. Inference decode is often limited by memory capacity and bandwidth: NVIDIA lists 4.8 TB/s and 141 GB for H200 SXM versus 3.35 TB/s and 80 GB for H100 SXM. For single-card prototypes, RTX PRO 6000 Blackwell offers 96 GB.

Test it first. This guide quotes no performance percentages for reduced-spec cards because NVIDIA does not publish full datasheets for them on its data center pages. Ask the supplier for the exact SKU datasheet and run your own workload.

A 70B model needs about 140 GB for weights at FP16 (70 billion x 2 bytes) and about 35 GB at INT4 (70 billion x 0.5 byte), before KV cache and overhead. A 96 GB card such as RTX PRO 6000 Blackwell can hold a 4-bit quantized 70B model; it cannot hold the FP16 weights.

Options include reduced-specification NVIDIA cards and domestic accelerators such as Huawei Ascend. Availability follows export policy and changes over time, so verify current status, datasheets and supply in writing before procurement.

Buy when utilization stays high for years and you have an operations team; rent for short experiments or low utilization. Compare current written cloud rates and a supplier quotation (industry estimate, to be confirmed) for your own workload rather than relying on a generic breakeven.

Key Takeaway

Key Takeaway: Pick the bottleneck before the card. Training needs compute and NVLink scale-up; inference needs memory capacity and bandwidth; a proof of concept needs capacity on one card such as the 96 GB RTX PRO 6000. BENQUAN GLOBAL LIMITED supplies NVIDIA GPU servers and RTX PRO 6000 workstations through authorized channels.

Not sure your compute BOM is optimal?

If GPU selection, cluster sizing, memory requirements, or compliance status is unclear, click below for a free compatibility and delivery risk assessment. Fast global BOM validation and risk assessment.

Free Compatibility & Delivery Risk Assessment

Or call +852-21349834 · info@benquan.hk

Contents