SambaNova · RDU · Layer 2
SambaNova SN40L
Serving many large models from one system. The three-tier memory design exists to swap models in and out rather than to hold one resident.
rent Rentable by the hour from multiple clouds; purchasable at scale.
Specifications
- Memory
- 520 MiB SRAM + 64 GiB HBM + up to 1.5 TiB DDR per socket
- Bandwidth
- Over 1 TB/s DDR-to-HBM model streaming
- Compute
- Reconfigurable dataflow
- Power
- Not published
- Interconnect
- Socket-to-socket fabric
- Software
- SambaFlow
- Form factor
- accelerator
- Workload
- inference
Verified 2026-09-06. Fields reading “not published” are exactly that — we do not estimate a figure a vendor withholds.
The catch
The most unusual architecture here, so the least portable. Independently verified at 1,084 tokens/s for Llama3-8B — but on 16 sockets, which is the number that matters.
The economics
Its argument is models-per-dollar rather than tokens-per-dollar. If you serve one model, this is the wrong shape.
Cost per million tokens is the metric that decides most real purchases, and there is no neutral benchmark for it — every published figure comes from a company selling one side of the comparison. Read all of them, ours included, as claims rather than measurements.
The power question underneath this
A rack of frontier accelerators draws 120–200 kW against a 2026 average of about 27 kW, and the US grid interconnection queue exceeds 2,600 GW with waits approaching five years. Whether you can energise this part is now a harder question than whether you can buy it.
Layer 1 — Energy · The interconnection queue · Rack power density · Tokens per watt
Compared against
- SambaNova SN40L vs Groq LPU — Many models on one system against one model served cheaply.Compare with Groq LPU
Related silicon
- Cerebras WSE-3 — 44 GB on-chip SRAM, rent
- Groq LPU — 230 MB SRAM per chip, no HBM, rent
- AMD Instinct MI300X — 192 GB HBM3, rent
- AMD Instinct MI350X — 288 GB HBM3E, rent
- AMD Instinct MI355X — 288 GB HBM3E, rent
- AWS Inferentia 2 — Not published in directly comparable terms, cloud only