Layer 4 · self-hosting reality check
What it actually takes to self-host vLLM
The docs say 16 GB system RAM. In practice you want 24 GB VRAM minimum for useful production serving. Here is the honest version — real requirements, real monthly cost, what you will be maintaining, and the one thing that catches people out.
Usually reached from OpenAI API (ChatGPT) alternatives, where vLLM is one of the picks.
Wondering whether you need to at all? Is the OpenAI API free? — what the free tier actually allows, and where the wall is.
| RAM — documented minimum | 16 GB system RAM |
|---|---|
| RAM — what it really needs | 24 GB VRAM minimum for useful production serving |
| CPU | 8 vCPU alongside the GPU |
| Disk | Model weights — 15 GB for a 7B in fp16, more for larger |
| Monthly cost | $300–900/mo for a rented A100 or L40S, against managed inference with a platform margin on every token |
| Setup time | A day including CUDA |
| How you install it | pip install plus `vllm serve <model>`; it exposes an OpenAI-compatible endpoint on port 8000 |
| Ongoing maintenance | Moderate. CUDA and driver versions are the recurring pain, not vLLM itself. |
| Where it stops scaling | Thousands of concurrent requests per GPU with continuous batching. This is the serving engine behind a large share of the providers you would otherwise pay. |
The thing that catches people out
It pre-allocates almost all GPU memory on startup by design — that is PagedAttention working correctly, not a leak. But it means nothing else can share the card, and `gpu_memory_utilization` must be lowered if you want to co-locate anything. People see 95% VRAM used at idle and assume something is broken.
When not to self-host vLLM
Your traffic is spiky or unproven. A GPU bills whether or not anyone calls it, and below roughly 30% utilisation a managed endpoint wins on cost.
Every guide here carries this section. A site that only ever tells you to self-host is selling something — the useful answer is sometimes no.
Other Layer 4 self-hosting guides
- Self-hosting Ollama8 GB VRAM for a 7B model at usable speed; 24 GB for 30B-class
- Self-hosting QdrantVectors × dimensions × 4 bytes, in RAM, plus overhead — 1M × 768d is roughly 3 GB
- Self-hosting pgvector8 GB — the HNSW index wants to be resident
- Self-hosting LlamaIndex4 GB for the app; your vector store is the real cost
- Self-hosting faster-whisper5 GB VRAM for large-v3 in float16; 2 GB with int8
- Self-hosting Langfuse4 GB
Common questions
- How much RAM does vLLM actually need?
- 24 GB VRAM minimum for useful production serving in practice. The documented minimum is 16 GB system RAM, which is the figure at which the process starts rather than the figure at which it works under real use. 8 vCPU alongside the GPU alongside it.
- What does self-hosting vLLM cost per month?
- $300–900/mo for a rented A100 or L40S, against managed inference with a platform margin on every token This is commodity VPS pricing and excludes your time, which is the larger cost for most people — budget for moderate. CUDA and driver versions are the recurring pain, not vLLM itself.
- How long does it take to set up vLLM?
- A day including CUDA, via pip install plus `vllm serve <model>`; it exposes an OpenAI-compatible endpoint on port 8000.
- When should I NOT self-host vLLM?
- Your traffic is spiky or unproven. A GPU bills whether or not anyone calls it, and below roughly 30% utilisation a managed endpoint wins on cost.
- What is the most common mistake when self-hosting vLLM?
- It pre-allocates almost all GPU memory on startup by design — that is PagedAttention working correctly, not a leak. But it means nothing else can share the card, and `gpu_memory_utilization` must be lowered if you want to co-locate anything. People see 95% VRAM used at idle and assume something is broken.