Layer 2 · head to head
Groq LPU vs NVIDIA H100
Specialist inference silicon against the general-purpose default.
| Groq LPU | NVIDIA H100 | |
|---|---|---|
| Can you get it | Rentable by the hour from multiple clouds; purchasable at scale. | Rentable by the hour from multiple clouds; purchasable at scale. |
| Memory | 230 MB SRAM per chip, no HBM | 80 GB HBM3 |
| Bandwidth | 80 TB/s on-chip | 3.35 TB/s |
| Compute | Deterministic inference architecture | Hopper-generation FP8 |
| Power | Not published per chip | 700 W |
| Interconnect | Chip-to-chip fabric | NVLink 4 |
| Software | GroqCloud API, GroqWare | CUDA |
| Workload | inference | both |
Groq LPU
For: Cheap, fast token generation via API. The lowest cost per million tokens among the specialist inference parts.
The catch: 230 MB per chip means the model is split across many chips — Llama2-70B at Groq's headline speed required 576 chips, roughly eight racks. The per-chip spec is misleading unless you read it as a per-cluster architecture.
Economics: Cheaper per million tokens than Cerebras while being materially slower. If your product is priced per token rather than per millisecond, that is the right trade.
NVIDIA H100
For: The reference point everything else is benchmarked against, and still the most rentable accelerator on earth.
The catch: 80 GB is the binding limit. Large models need multi-GPU sharding that a 141 GB or 288 GB part would not, and sharding costs you latency and complexity.
Economics: The benchmark denominator. When a vendor claims '2.6x an H100', this is the H100 they mean.
Before either — can you power it?
Not published per chip against 700 W. In 2026 that comparison usually matters more than the FLOPS one: the US interconnection queue exceeds 2,600 GW with waits approaching five years, and roughly 80% of projects withdraw before energising.