Layer 4 · self-hosting reality check
What it actually takes to self-host Promptfoo
The docs say 512 MB. In practice you want 2 GB. Here is the honest version — real requirements, real monthly cost, what you will be maintaining, and the one thing that catches people out.
Usually reached from Braintrust alternatives, where Promptfoo is one of the picks.
Wondering whether you need to at all? Is Braintrust free? — what the free tier actually allows, and where the wall is.
| RAM — documented minimum | 512 MB |
|---|---|
| RAM — what it really needs | 2 GB |
| CPU | 2 vCPU |
| Disk | Small — results are files |
| Monthly cost | $0. It runs in CI on hardware you already pay for. |
| Setup time | 1 hour for the first ten test cases |
| How you install it | npm install -g promptfoo; tests are YAML in your repository |
| Ongoing maintenance | Low. The suite grows with your prompts. |
| Where it stops scaling | Thousands of test cases. Your model bill is the constraint, not the tool. |
The thing that catches people out
LLM-as-judge assertions call a model on every test case, so a large suite running on every commit costs real money — occasionally more than the platform you left. Split it: cheap deterministic assertions on every pull request, the judged suite nightly. Pointing the judge at a locally served model takes that cost to near zero.
When not to self-host Promptfoo
You need a hosted dashboard a product manager will open. Promptfoo's reporting is local and developer-shaped.
Every guide here carries this section. A site that only ever tells you to self-host is selling something — the useful answer is sometimes no.
Other Layer 4 self-hosting guides
- Self-hosting Ollama8 GB VRAM for a 7B model at usable speed; 24 GB for 30B-class
- Self-hosting vLLM24 GB VRAM minimum for useful production serving
- Self-hosting QdrantVectors × dimensions × 4 bytes, in RAM, plus overhead — 1M × 768d is roughly 3 GB
- Self-hosting pgvector8 GB — the HNSW index wants to be resident
- Self-hosting LlamaIndex4 GB for the app; your vector store is the real cost
- Self-hosting faster-whisper5 GB VRAM for large-v3 in float16; 2 GB with int8
Common questions
- How much RAM does Promptfoo actually need?
- 2 GB in practice. The documented minimum is 512 MB, which is the figure at which the process starts rather than the figure at which it works under real use. 2 vCPU alongside it.
- What does self-hosting Promptfoo cost per month?
- $0. It runs in CI on hardware you already pay for. This is commodity VPS pricing and excludes your time, which is the larger cost for most people — budget for low. The suite grows with your prompts.
- How long does it take to set up Promptfoo?
- 1 hour for the first ten test cases, via npm install -g promptfoo; tests are YAML in your repository.
- When should I NOT self-host Promptfoo?
- You need a hosted dashboard a product manager will open. Promptfoo's reporting is local and developer-shaped.
- What is the most common mistake when self-hosting Promptfoo?
- LLM-as-judge assertions call a model on every test case, so a large suite running on every commit costs real money — occasionally more than the platform you left. Split it: cheap deterministic assertions on every pull request, the judged suite nightly. Pointing the judge at a locally served model takes that cost to near zero.