Layer 4 · self-hosting reality check
What it actually takes to self-host Ollama
The docs say 8 GB system RAM. In practice you want 8 GB VRAM for a 7B model at usable speed; 24 GB for 30B-class. Here is the honest version — real requirements, real monthly cost, what you will be maintaining, and the one thing that catches people out.
Usually reached from OpenAI API (ChatGPT) alternatives, where Ollama is one of the picks.
| RAM — documented minimum | 8 GB system RAM |
|---|---|
| RAM — what it really needs | 8 GB VRAM for a 7B model at usable speed; 24 GB for 30B-class |
| CPU | Any modern CPU; the GPU is what matters |
| Disk | 4–5 GB per 7B model, 40 GB+ for the large ones. Model files add up fast. |
| Monthly cost | $0 on hardware you own. $150–400/mo for a rented 24 GB GPU, against per-token API billing that is cheaper below roughly 2M tokens a month. |
| Setup time | 10 minutes |
| How you install it | one installer; `ollama run llama3` downloads and serves in a single command |
| Ongoing maintenance | Very low. Model updates are a pull. |
| Where it stops scaling | Single-user or small-team use. For concurrent production traffic you want vLLM, which is built for throughput rather than convenience. |
The thing that catches people out
CPU-only inference works and is roughly ten times too slow to use. People install it on a VPS with no GPU, get four tokens a second, and conclude local models are not viable. They are — on a GPU. Check VRAM before anything else, and if the model does not fit, Ollama silently offloads layers to system RAM and the speed collapses without an error.
When not to self-host Ollama
Your volume is low and spiky. Below a couple of million tokens a month, the API is genuinely cheaper than a GPU sitting idle — the maths only flips at steady load.
Every guide here carries this section. A site that only ever tells you to self-host is selling something — the useful answer is sometimes no.
Other Layer 4 self-hosting guides
- Self-hosting vLLM24 GB VRAM minimum for useful production serving
- Self-hosting QdrantVectors × dimensions × 4 bytes, in RAM, plus overhead — 1M × 768d is roughly 3 GB
- Self-hosting pgvector8 GB — the HNSW index wants to be resident
- Self-hosting LlamaIndex4 GB for the app; your vector store is the real cost
- Self-hosting faster-whisper5 GB VRAM for large-v3 in float16; 2 GB with int8
- Self-hosting Langfuse4 GB
Common questions
- How much RAM does Ollama actually need?
- 8 GB VRAM for a 7B model at usable speed; 24 GB for 30B-class in practice. The documented minimum is 8 GB system RAM, which is the figure at which the process starts rather than the figure at which it works under real use. Any modern CPU; the GPU is what matters alongside it.
- What does self-hosting Ollama cost per month?
- $0 on hardware you own. $150–400/mo for a rented 24 GB GPU, against per-token API billing that is cheaper below roughly 2M tokens a month. This is commodity VPS pricing and excludes your time, which is the larger cost for most people — budget for very low. Model updates are a pull.
- How long does it take to set up Ollama?
- 10 minutes, via one installer; `ollama run llama3` downloads and serves in a single command.
- When should I NOT self-host Ollama?
- Your volume is low and spiky. Below a couple of million tokens a month, the API is genuinely cheaper than a GPU sitting idle — the maths only flips at steady load.
- What is the most common mistake when self-hosting Ollama?
- CPU-only inference works and is roughly ten times too slow to use. People install it on a VPS with no GPU, get four tokens a second, and conclude local models are not viable. They are — on a GPU. Check VRAM before anything else, and if the model does not fit, Ollama silently offloads layers to system RAM and the speed collapses without an error.