Layer 4 · self-hosting reality check
What it actually takes to self-host Promptfoo
The docs say 512 MB. In practice you want 2 GB. Here is the honest version — real requirements, real monthly cost, what you will be maintaining, and the one thing that catches people out.
Usually reached from Braintrust alternatives, where Promptfoo is one of the picks.
| RAM — documented minimum | 512 MB |
|---|---|
| RAM — what it really needs | 2 GB |
| CPU | 2 vCPU |
| Disk | Small — results are files |
| Monthly cost | $0. It runs in CI on hardware you already pay for. |
| Setup time | 1 hour for the first ten test cases |
| How you install it | npm install -g promptfoo; tests are YAML in your repository |
| Ongoing maintenance | Low. The suite grows with your prompts. |
| Where it stops scaling | Thousands of test cases. Your model bill is the constraint, not the tool. |
The thing that catches people out
LLM-as-judge assertions call a model on every test case, so a large suite running on every commit costs real money — occasionally more than the platform you left. Split it: cheap deterministic assertions on every pull request, the judged suite nightly. Pointing the judge at a locally served model takes that cost to near zero.
When not to self-host Promptfoo
You need a hosted dashboard a product manager will open. Promptfoo's reporting is local and developer-shaped.
Every guide here carries this section. A site that only ever tells you to self-host is selling something — the useful answer is sometimes no.
Other Layer 4 self-hosting guides
- Self-hosting Ollama8 GB VRAM for a 7B model at usable speed; 24 GB for 30B-class
- Self-hosting vLLM24 GB VRAM minimum for useful production serving
- Self-hosting QdrantVectors × dimensions × 4 bytes, in RAM, plus overhead — 1M × 768d is roughly 3 GB
- Self-hosting pgvector8 GB — the HNSW index wants to be resident
- Self-hosting LlamaIndex4 GB for the app; your vector store is the real cost
- Self-hosting faster-whisper5 GB VRAM for large-v3 in float16; 2 GB with int8
Common questions
- How much RAM does Promptfoo actually need?
- 2 GB in practice. The documented minimum is 512 MB, which is the figure at which the process starts rather than the figure at which it works under real use. 2 vCPU alongside it.
- What does self-hosting Promptfoo cost per month?
- $0. It runs in CI on hardware you already pay for. This is commodity VPS pricing and excludes your time, which is the larger cost for most people — budget for low. The suite grows with your prompts.
- How long does it take to set up Promptfoo?
- 1 hour for the first ten test cases, via npm install -g promptfoo; tests are YAML in your repository.
- When should I NOT self-host Promptfoo?
- You need a hosted dashboard a product manager will open. Promptfoo's reporting is local and developer-shaped.
- What is the most common mistake when self-hosting Promptfoo?
- LLM-as-judge assertions call a model on every test case, so a large suite running on every commit costs real money — occasionally more than the platform you left. Split it: cheap deterministic assertions on every pull request, the judged suite nightly. Pointing the judge at a locally served model takes that cost to near zero.